00000800:00000100:0.0F:1584532877.141027:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584532877.141587:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0F:1584532877.153670:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584532871/real 1584532871] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584532958 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584532877.153680:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584532877.153682:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584532877.153684:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584532877.153700:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532877.153702:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532877.153719:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532877.153720:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532877.153721:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532877.153723:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584532877.153724:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584532877.153737:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14677 00000100:00080000:18.0:1584532877.153741:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584532877.153760:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532877.153761:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532877.153762:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532877.153764:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532877.153764:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532877.153765:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532877.153766:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532877.153768:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0F:1584532877.159997:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584532877.160005:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584532877.160007:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584532877.160008:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584532877.160010:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584532877.160012:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584532877.160014:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584532877.160016:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584532958 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584532877.160049:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078396688 new x1661492078396768 00000800:00020000:2.0:1584532883.916127:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584532883.916130:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584532883.916132:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584532883.916553:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584532883.928600:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584532877/real 1584532877] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584532964 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584532883.928610:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584532883.928611:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584532883.928613:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584532883.928621:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532883.928623:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532883.928624:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532883.928625:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532883.928626:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532883.928627:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584532883.928628:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584532883.928630:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14683 00000100:00080000:18.0:1584532883.928633:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584532883.928647:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532883.928648:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532883.928649:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532883.928650:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532883.928651:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532883.928652:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532883.928654:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532883.928655:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584532883.934532:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584532883.934538:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584532883.934540:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584532883.934541:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584532883.934543:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584532883.934545:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584532883.934546:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584532883.934548:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584532964 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584532883.934578:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078396768 new x1661492078396800 00000800:00000100:0.0:1584532889.140555:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584532889.141016:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584532889.153070:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584532883/real 1584532883] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584532970 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584532889.153078:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584532889.153080:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584532889.153081:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584532889.153087:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532889.153089:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532889.153089:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532889.153091:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532889.153091:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532889.153093:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584532889.153093:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584532889.153095:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14689 00000100:00080000:18.0:1584532889.153097:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584532889.153105:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532889.153107:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532889.153107:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532889.153109:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532889.153110:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532889.153112:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532889.153112:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532889.153113:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584532889.159052:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584532889.159056:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584532889.159057:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584532889.159058:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584532889.159060:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584532889.159061:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584532889.159063:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584532889.159065:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584532970 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584532889.159094:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078396800 new x1661492078396832 00000800:00000100:0.0:1584532895.140793:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584532895.141250:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584532895.153322:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584532889/real 1584532889] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584532976 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584532895.153330:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584532895.153331:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584532895.153332:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584532895.153338:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532895.153340:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532895.153340:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532895.153342:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532895.153342:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532895.153344:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584532895.153344:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584532895.153346:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14695 00000100:00080000:18.0:1584532895.153348:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584532895.153355:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532895.153356:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532895.153358:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532895.153359:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532895.153360:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532895.153361:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532895.153373:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532895.153374:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584532895.159606:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584532895.159610:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584532895.159611:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584532895.159612:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584532895.159614:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584532895.159616:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584532895.159617:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584532895.159619:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584532976 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584532895.159653:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078396832 new x1661492078396880 00000800:00000100:0.0:1584532901.141027:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584532901.141567:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584532901.153632:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584532895/real 1584532895] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584532982 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584532901.153640:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584532901.153641:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584532901.153642:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584532901.153656:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532901.153658:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532901.153663:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532901.153665:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532901.153665:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532901.153667:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584532901.153667:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584532901.153669:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14701 00000100:00080000:18.0:1584532901.153671:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584532901.153678:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532901.153679:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532901.153680:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532901.153681:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532901.153682:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532901.153683:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532901.153683:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532901.153685:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0F:1584532901.159603:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584532901.159608:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584532901.159609:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584532901.159610:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584532901.159611:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584532901.159613:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584532901.159615:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584532901.159616:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584532982 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584532901.159649:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078396880 new x1661492078396944 00000800:00020000:2.0:1584532907.916062:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584532907.916065:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584532907.916067:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584532907.916728:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584532907.928783:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584532901/real 1584532901] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584532988 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584532907.928791:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584532907.928792:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584532907.928793:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584532907.928800:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532907.928801:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532907.928802:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532907.928803:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532907.928804:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532907.928805:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584532907.928806:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584532907.928809:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14707 00000100:00080000:18.0:1584532907.928811:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584532907.928820:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532907.928821:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532907.928822:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532907.928823:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532907.928824:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532907.928825:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532907.928826:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532907.928827:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0F:1584532907.934866:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584532907.934871:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584532907.934872:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584532907.934873:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584532907.934874:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584532907.934876:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584532907.934878:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584532907.934880:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584532988 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584532907.934910:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078396944 new x1661492078396992 00000800:00000100:0.0:1584532913.141371:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584532913.141824:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584532913.153875:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584532907/real 1584532907] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584532994 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584532913.153883:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584532913.153884:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584532913.153885:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584532913.153891:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532913.153892:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532913.153893:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532913.153894:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532913.153895:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532913.153897:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584532913.153898:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584532913.153900:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14713 00000100:00080000:18.0:1584532913.153902:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584532913.153910:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532913.153911:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532913.153912:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532913.153913:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532913.153914:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532913.153915:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532913.153916:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532913.153917:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584532913.159660:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584532913.159664:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584532913.159665:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584532913.159666:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584532913.159667:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584532913.159669:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584532913.159671:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584532913.159672:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584532994 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584532913.159702:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078396992 new x1661492078397024 00000800:00000100:0.0:1584532919.141602:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584532919.142031:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584532919.154120:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584532913/real 1584532913] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533000 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584532919.154127:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584532919.154128:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584532919.154130:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584532919.154135:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532919.154137:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532919.154138:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532919.154140:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532919.154140:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532919.154141:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584532919.154142:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584532919.154144:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14719 00000100:00080000:18.0:1584532919.154145:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584532919.154153:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532919.154155:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532919.154155:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532919.154156:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532919.154157:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532919.154158:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532919.154159:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532919.154160:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0F:1584532919.160038:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584532919.160043:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584532919.160044:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584532919.160045:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584532919.160046:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584532919.160048:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584532919.160049:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584532919.160051:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533000 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584532919.160080:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078397024 new x1661492078397056 00000800:00000100:0.0:1584532925.141712:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584532925.142248:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584532925.154307:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584532919/real 1584532919] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533006 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584532925.154317:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584532925.154318:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584532925.154320:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584532925.154338:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532925.154341:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532925.154346:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532925.154348:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532925.154348:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532925.154350:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584532925.154351:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584532925.154353:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14725 00000100:00080000:18.0:1584532925.154356:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584532925.154363:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532925.154364:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532925.154365:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532925.154366:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532925.154367:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532925.154368:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584532925.154369:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584532925.154370:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584532925.160411:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584532925.160421:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584532925.160425:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584532925.160427:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584532925.160431:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584532925.160435:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584532925.160438:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584532925.160443:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533006 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0F:1584532925.160461:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078397056 new x1661492078397136 00000800:00020000:2.0:1584532931.916043:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584532931.916046:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584532931.916049:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584532931.916467:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584532931.928553:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584532925/real 1584532925] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533012 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584532931.928564:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584532931.928565:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584532931.928567:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584532931.928575:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532931.928577:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532931.928578:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532931.928579:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532931.928580:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532931.928581:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584532931.928582:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584532931.928584:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14731 00000100:00080000:20.0:1584532931.928587:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584532931.928603:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532931.928605:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532931.928605:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532931.928607:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532931.928607:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532931.928608:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532931.928609:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532931.928611:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584532931.934655:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584532931.934661:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584532931.934663:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584532931.934664:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584532931.934666:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584532931.934668:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584532931.934669:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584532931.934671:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533012 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584532931.934700:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078397136 new x1661492078397168 00000800:00000100:0.0:1584532937.140278:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584532937.140721:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584532937.152792:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584532931/real 1584532931] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533018 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584532937.196564:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584532937.196565:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584532937.196567:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584532937.196573:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532937.196576:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532937.196576:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532937.196578:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532937.196578:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532937.196591:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584532937.196592:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584532937.196594:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14737 00000100:00080000:20.0:1584532937.196597:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584532937.196608:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532937.196610:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532937.196611:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532937.196612:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532937.196613:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532937.196615:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532937.196616:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532937.196617:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584532937.202837:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584532937.202845:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584532937.202847:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584532937.202848:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584532937.202850:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584532937.202853:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584532937.202854:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584532937.202856:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533018 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584532937.202885:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078397168 new x1661492078397216 00000800:00000100:0.0:1584532943.140095:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584532943.140506:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584532943.152556:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584532937/real 1584532937] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533024 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584532943.152564:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584532943.152565:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584532943.152567:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584532943.152572:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532943.152573:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532943.152574:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532943.152575:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532943.152576:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532943.152577:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584532943.152578:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584532943.152579:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14743 00000100:00080000:20.0:1584532943.152581:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584532943.152590:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532943.152592:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532943.152592:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532943.152595:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532943.152595:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532943.152596:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532943.152597:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532943.152598:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584532943.158358:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584532943.158362:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584532943.158364:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584532943.158364:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584532943.158366:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584532943.158368:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584532943.158369:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584532943.158371:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533024 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584532943.158400:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078397216 new x1661492078397248 00000800:00000100:0.0:1584532949.140406:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584532949.140869:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584532949.152923:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584532943/real 1584532943] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533030 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584532949.152932:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584532949.152933:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584532949.152934:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584532949.152948:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532949.152950:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532949.152955:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532949.152956:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532949.152957:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532949.152959:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584532949.152959:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584532949.152962:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14749 00000100:00080000:20.0:1584532949.152964:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584532949.152970:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532949.152972:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532949.152973:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532949.152974:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532949.152975:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532949.152976:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532949.152976:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532949.152978:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0F:1584532949.158878:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584532949.158883:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584532949.158885:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584532949.158886:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584532949.158887:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584532949.158889:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584532949.158891:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584532949.158893:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533030 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584532949.158921:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078397248 new x1661492078397312 00000800:00020000:4.0:1584532955.916054:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584532955.916057:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584532955.916059:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584532955.916438:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584532955.928490:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584532949/real 1584532949] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533036 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584532955.928498:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584532955.928500:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584532955.928501:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584532955.928506:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532955.928508:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532955.928509:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532955.928510:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532955.928511:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532955.928512:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584532955.928513:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584532955.928515:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14755 00000100:00080000:20.0:1584532955.928518:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584532955.928527:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532955.928528:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532955.928529:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532955.928530:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532955.928530:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532955.928532:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532955.928532:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532955.928533:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584532955.934627:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584532955.934631:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584532955.934633:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584532955.934634:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584532955.934635:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584532955.934637:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584532955.934639:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584532955.934641:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533036 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584532955.934649:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078397312 new x1661492078397360 00000800:00000100:0.0:1584532961.140988:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584532961.141487:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584532961.153540:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584532955/real 1584532955] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533042 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584532961.153548:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584532961.153549:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584532961.153550:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584532961.153555:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532961.153557:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532961.153557:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532961.153559:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532961.153559:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532961.153561:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584532961.153562:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584532961.153564:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14761 00000100:00080000:20.0:1584532961.153566:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584532961.153574:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532961.153575:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532961.153576:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532961.153577:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532961.153578:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532961.153579:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532961.153580:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532961.153581:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584532961.159328:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584532961.159331:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584532961.159333:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584532961.159334:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584532961.159335:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584532961.159337:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584532961.159338:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584532961.159340:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533042 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584532961.159369:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078397360 new x1661492078397392 00000800:00000100:0.0:1584532967.141159:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584532967.141601:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584532967.153668:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584532961/real 1584532961] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533048 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584532967.153676:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584532967.153677:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584532967.153678:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584532967.153684:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532967.153686:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532967.153686:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532967.153687:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532967.153689:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532967.153691:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584532967.153691:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584532967.153693:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14767 00000100:00080000:20.0:1584532967.153695:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584532967.153703:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532967.153704:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532967.153705:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532967.153706:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532967.153706:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532967.153707:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532967.153708:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532967.153709:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584532967.159694:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584532967.159700:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584532967.159702:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584532967.159703:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584532967.159704:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584532967.159706:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584532967.159708:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584532967.159710:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533048 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584532967.159739:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078397392 new x1661492078397440 00000800:00000100:0.0:1584532973.141478:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584532973.141940:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584532973.153991:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584532967/real 1584532967] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533054 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584532973.154000:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584532973.154001:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584532973.154002:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584532973.154016:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532973.154018:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532973.154024:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532973.154026:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532973.154038:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532973.154039:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584532973.154040:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584532973.154042:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14773 00000100:00080000:20.0:1584532973.154044:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584532973.154050:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532973.154051:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532973.154052:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532973.154053:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532973.154054:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532973.154055:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532973.154055:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532973.154056:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584532973.159702:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584532973.159706:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584532973.159707:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584532973.159708:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584532973.159710:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584532973.159711:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584532973.159713:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584532973.159714:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533054 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584532973.159743:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078397440 new x1661492078397504 00000800:00020000:4.0:1584532979.916062:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584532979.916065:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584532979.916067:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584532979.916502:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584532979.928580:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584532973/real 1584532973] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533060 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584532979.928588:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584532979.928589:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584532979.928591:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584532979.928597:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532979.928599:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532979.928600:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532979.928601:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532979.928601:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532979.928603:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584532979.928603:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584532979.928605:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14779 00000100:00080000:20.0:1584532979.928607:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584532979.928615:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532979.928616:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532979.928617:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532979.928618:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532979.928619:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532979.928620:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532979.928620:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532979.928622:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584532979.934514:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584532979.934518:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584532979.934519:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584532979.934520:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584532979.934521:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584532979.934523:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584532979.934524:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584532979.934526:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533060 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584532979.934556:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078397504 new x1661492078397536 00000800:00000100:0.0:1584532985.141954:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584532985.142356:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584532985.154426:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584532979/real 1584532979] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533066 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584532985.154437:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584532985.154440:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584532985.154442:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584532985.154449:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532985.154452:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532985.154452:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532985.154454:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532985.154454:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532985.154456:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584532985.154457:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584532985.154459:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14785 00000100:00080000:20.0:1584532985.154463:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584532985.154477:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532985.154478:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532985.154479:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532985.154480:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532985.154481:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532985.154482:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532985.154483:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532985.154484:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584532985.160627:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584532985.160635:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584532985.160637:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584532985.160638:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584532985.160640:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584532985.160643:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584532985.160644:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584532985.160646:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533066 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584532985.160676:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078397536 new x1661492078397584 00000800:00000100:0.0:1584532991.142177:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584532991.142662:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584532991.154719:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584532985/real 1584532985] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533072 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584532991.154729:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584532991.154730:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584532991.154732:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584532991.154738:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532991.154741:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532991.154741:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532991.154743:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532991.154743:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532991.154745:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584532991.154746:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584532991.154748:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14791 00000100:00080000:20.0:1584532991.154750:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584532991.154764:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532991.154766:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532991.154767:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532991.154768:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532991.154768:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532991.154770:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532991.154770:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532991.154772:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584532991.160759:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584532991.160765:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584532991.160768:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584532991.160769:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584532991.160770:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584532991.160773:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584532991.160774:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584532991.160776:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533072 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584532991.160807:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078397584 new x1661492078397616 00000800:00000100:0.0:1584532997.142388:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584532997.142863:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584532997.154937:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584532991/real 1584532991] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533078 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584532997.154947:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584532997.154948:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584532997.154950:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584532997.154971:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532997.154973:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532997.154989:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532997.154990:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532997.154991:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532997.154992:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584532997.154993:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584532997.154995:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14797 00000100:00080000:20.0:1584532997.155009:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584532997.155035:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532997.155036:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532997.155037:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532997.155038:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532997.155039:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532997.155041:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584532997.155042:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584532997.155044:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584532997.161059:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584532997.161065:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584532997.161067:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584532997.161069:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584532997.161071:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584532997.161073:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584532997.161074:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584532997.161076:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533078 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584532997.161106:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078397616 new x1661492078397696 00000800:00020000:2.0:1584533003.916043:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533003.916047:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533003.916049:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533003.916459:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584533003.928509:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584532997/real 1584532997] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533084 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584533003.928521:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584533003.928522:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584533003.928524:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584533003.928533:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533003.928535:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533003.928536:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533003.928538:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533003.928538:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533003.928540:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584533003.928541:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584533003.928543:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14803 00000100:00080000:20.0:1584533003.928546:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584533003.928563:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533003.928564:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533003.928565:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533003.928566:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533003.928568:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533003.928569:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533003.928570:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533003.928571:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533003.934503:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584533003.934511:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584533003.934513:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584533003.934514:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584533003.934516:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584533003.934519:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584533003.934520:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584533003.934522:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533084 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584533003.934531:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078397696 new x1661492078397728 00000800:00000100:0.0:1584533009.139417:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533009.139884:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584533009.151934:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533003/real 1584533003] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533090 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584533009.151942:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584533009.177284:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584533009.177285:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584533009.177289:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533009.177291:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533009.177292:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533009.177293:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533009.177293:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533009.177295:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584533009.177296:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584533009.177297:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14809 00000100:00080000:20.0:1584533009.177299:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584533009.177308:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533009.177309:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533009.177311:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533009.177312:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533009.177313:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533009.177314:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533009.177315:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533009.177316:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533009.183239:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584533009.183243:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584533009.183245:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584533009.183245:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584533009.183247:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584533009.183249:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584533009.183250:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584533009.183252:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533090 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584533009.183281:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078397728 new x1661492078397760 00000800:00000100:0.0:1584533015.139096:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533015.139519:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584533015.151569:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533009/real 1584533009] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533096 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584533015.151577:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584533015.151579:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584533015.151580:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584533015.151585:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533015.151587:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533015.151587:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533015.151589:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533015.151589:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533015.151591:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584533015.151591:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584533015.151593:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14815 00000100:00080000:20.0:1584533015.151595:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584533015.151605:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533015.151607:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533015.151607:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533015.151609:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533015.151609:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533015.151610:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533015.151611:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533015.151612:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533015.157516:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584533015.157520:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584533015.157521:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584533015.157522:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584533015.157523:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584533015.157525:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584533015.157527:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584533015.175313:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533096 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584533015.175344:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078397760 new x1661492078397808 00000800:00000100:0.0:1584533021.139432:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533021.139940:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584533021.151990:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533015/real 1584533015] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533102 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584533021.151998:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584533021.151999:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584533021.152000:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584533021.152015:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533021.152017:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533021.152023:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533021.152024:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533021.152024:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533021.152026:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584533021.152037:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584533021.152040:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14821 00000100:00080000:20.0:1584533021.152042:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584533021.152048:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533021.152049:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533021.152050:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533021.152051:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533021.152051:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533021.152052:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533021.152053:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533021.152054:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0F:1584533021.157921:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533021.157926:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533021.157927:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533021.157928:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533021.157929:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533021.157931:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533021.157933:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533021.157934:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533102 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584533021.157943:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078397808 new x1661492078397872 00000800:00020000:4.0:1584533027.916097:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533027.916100:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533027.916102:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533027.916522:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584533027.928601:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533021/real 1584533021] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533108 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584533027.928609:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584533027.928611:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584533027.928612:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584533027.928618:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533027.928620:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533027.928621:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533027.928622:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533027.928623:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533027.928625:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584533027.928626:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584533027.928628:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14827 00000100:00080000:20.0:1584533027.928630:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584533027.928639:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533027.928641:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533027.928642:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533027.928643:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533027.928643:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533027.928644:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533027.928645:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533027.928646:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533027.934550:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533027.934554:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533027.934556:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533027.934557:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533027.934559:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533027.934561:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533027.934562:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533027.934564:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533108 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584533027.934593:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078397872 new x1661492078397920 00000800:00000100:2.0:1584533033.139838:0:4532:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533033.140366:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584533033.152422:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533027/real 1584533027] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533114 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584533033.152430:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584533033.152431:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584533033.152432:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584533033.152437:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533033.152439:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533033.152440:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533033.152442:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533033.152442:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533033.152444:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584533033.152444:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584533033.152446:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14833 00000100:00080000:20.0:1584533033.152448:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584533033.152457:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533033.152458:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533033.152458:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533033.152459:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533033.152460:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533033.152461:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533033.152462:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533033.152463:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533033.158246:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533033.158249:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533033.158251:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533033.158252:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533033.158253:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533033.158255:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533033.158256:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533033.158258:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533114 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584533033.158268:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078397920 new x1661492078397952 00000800:00000100:0.0:1584533039.140067:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533039.140508:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584533039.152564:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533033/real 1584533033] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533120 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584533039.152572:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584533039.152573:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584533039.152574:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584533039.152580:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533039.152582:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533039.152583:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533039.152584:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533039.152585:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533039.152586:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584533039.152586:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584533039.152588:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14839 00000100:00080000:20.0:1584533039.152590:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584533039.152598:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533039.152599:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533039.152600:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533039.152601:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533039.152602:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533039.152603:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533039.152603:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533039.152605:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533039.199182:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533039.199186:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533039.199188:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533039.199189:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533039.199190:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533039.199192:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533039.199193:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533039.199195:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533120 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584533039.199224:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078397952 new x1661492078397984 00000800:00000100:0.0:1584533045.140381:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533045.140808:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584533045.152864:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533039/real 1584533039] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533126 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584533045.152875:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584533045.152877:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584533045.152880:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584533045.152904:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533045.152906:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533045.152911:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533045.152913:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533045.152913:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533045.152915:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584533045.152916:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584533045.152918:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14845 00000100:00080000:20.0:1584533045.152922:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584533045.152930:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533045.152931:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533045.152932:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533045.152933:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533045.152934:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533045.152935:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533045.152936:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533045.152937:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533045.188199:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533045.188206:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533045.188208:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533045.188209:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533045.188211:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533045.188214:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533045.188215:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533045.188217:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533126 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584533045.188226:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078397984 new x1661492078398064 00000800:00020000:2.0:1584533051.916083:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533051.916086:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533051.916089:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533051.916613:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584533051.928695:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533045/real 1584533045] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533132 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584533051.928711:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584533051.928712:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584533051.928714:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584533051.928724:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533051.928726:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533051.928726:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533051.928728:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533051.928728:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533051.928730:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584533051.928731:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584533051.928733:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14851 00000100:00080000:20.0:1584533051.928736:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584533051.928754:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533051.928755:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533051.928756:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533051.928757:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533051.928758:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533051.928759:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533051.928760:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533051.928761:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533051.984770:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533051.984777:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533051.984780:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533051.984781:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533051.984783:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533051.984785:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533051.984787:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533051.984788:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533132 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584533051.984818:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078398064 new x1661492078398096 00000800:00000100:0.0:1584533057.141272:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533057.141699:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584533057.153755:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533051/real 1584533051] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533138 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584533057.153764:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584533057.153765:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584533057.153766:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584533057.153772:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533057.153774:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533057.153775:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533057.153776:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533057.153777:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533057.153778:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584533057.153779:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584533057.153781:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14857 00000100:00080000:20.0:1584533057.153783:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584533057.153793:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533057.153794:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533057.153795:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533057.153796:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533057.153797:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533057.153798:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533057.153800:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533057.153801:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533057.201011:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533057.201017:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533057.201019:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533057.201020:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533057.201021:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533057.201023:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533057.201025:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533057.201034:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533138 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584533057.201045:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078398096 new x1661492078398144 00000800:00000100:0.0:1584533063.141686:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533063.142218:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584533063.154267:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533057/real 1584533057] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533144 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584533063.154275:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584533063.154276:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584533063.154278:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584533063.154283:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533063.154284:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533063.154285:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533063.154286:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533063.154287:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533063.154288:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584533063.154289:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584533063.154291:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14863 00000100:00080000:20.0:1584533063.154293:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584533063.154301:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533063.154302:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533063.154303:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533063.154304:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533063.154305:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533063.154307:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533063.154307:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533063.154309:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533063.199873:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533063.199877:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533063.199878:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533063.199879:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533063.199880:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533063.199882:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533063.199884:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533063.199885:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533144 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584533063.199915:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078398144 new x1661492078398176 00000800:00000100:0.0:1584533069.141608:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533069.142060:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584533069.154120:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533063/real 1584533063] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533150 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584533069.154131:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584533069.154133:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584533069.154135:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584533069.154157:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533069.154159:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533069.154164:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533069.154166:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533069.154166:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533069.154168:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584533069.154169:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584533069.154172:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14869 00000100:00080000:20.0:1584533069.154175:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584533069.154182:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533069.154184:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533069.154185:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533069.154187:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533069.154187:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533069.154189:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533069.154189:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533069.154191:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533069.208623:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533069.208630:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533069.208632:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533069.208634:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533069.208636:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533069.208638:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533069.208639:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533069.208641:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533150 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0F:1584533069.208650:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078398176 new x1661492078398240 00000800:00000100:0.0:1584533075.141751:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533075.142187:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584533075.154256:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533069/real 1584533069] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533156 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584533075.154264:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584533075.154265:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584533075.154267:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584533075.154273:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533075.154274:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533075.154275:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533075.154276:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533075.154277:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533075.154278:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584533075.154279:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584533075.154280:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14875 00000100:00080000:22.0:1584533075.154282:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584533075.154294:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533075.154295:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533075.154295:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533075.154297:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533075.154297:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533075.154298:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533075.154299:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533075.154312:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533075.203406:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533075.203413:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533075.203415:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533075.203416:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533075.203418:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533075.203420:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533075.203421:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533075.203423:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533156 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584533075.203432:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078398240 new x1661492078398288 00000800:00000100:0.0:1584533081.141818:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533081.142277:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584533081.154327:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533075/real 1584533075] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533162 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584533081.154335:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584533081.154336:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584533081.154337:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584533081.154343:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533081.154345:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533081.154345:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533081.154347:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533081.154347:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533081.154349:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584533081.154349:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584533081.154351:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14881 00000100:00080000:22.0:1584533081.154353:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584533081.154361:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533081.154362:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533081.154363:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533081.154364:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533081.154365:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533081.154367:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533081.154367:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533081.154369:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533085.346523:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533085.346528:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533085.346530:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533085.346530:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533085.346532:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533085.346534:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533085.346535:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533085.346537:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533162 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584533085.346547:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078398288 new x1661492078398320 00000800:00020000:2.0:1584533090.916096:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533090.916102:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533090.916107:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533090.916209:0:25717:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584533090.928108:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533085/real 1584533085] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533175 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584533090.928117:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584533090.928118:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584533090.928119:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584533090.928135:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533090.928136:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533090.928142:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533090.928143:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533090.928144:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533090.928145:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584533090.928146:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584533090.928147:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14887 00000100:00080000:22.0:1584533090.928149:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584533090.928155:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533090.928156:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533090.928157:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533090.928159:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533090.928160:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533090.928161:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533090.928162:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533090.928163:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533090.977769:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533090.977774:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533090.977776:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533090.977776:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533090.977778:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533090.977780:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533090.977782:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533090.977783:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533175 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:0.0:1584533090.977793:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078398320 new x1661492078398400 00000800:00020000:2.0:1584533096.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533096.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533096.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533096.916490:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584533096.928566:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533090/real 1584533090] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533177 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584533096.928574:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584533096.928575:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584533096.928577:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584533096.928583:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533096.928584:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533096.928585:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533096.928586:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533096.928587:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533096.928588:0:4532:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584533096.928589:0:4532:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584533096.928591:0:4532:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14896 00000100:00080000:2.0:1584533096.928593:0:4532:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584533096.928602:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533096.928604:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533096.928604:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533096.928605:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533096.928606:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533096.928607:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533096.928609:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533096.928610:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533096.979949:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533096.979953:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533096.979955:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533096.979956:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533096.979957:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533096.979959:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533096.979960:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533096.979962:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533177 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584533096.979971:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078398400 new x1661492078398432 00000800:00000100:0.0:1584533103.142525:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533103.142957:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584533103.155038:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533096/real 1584533096] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533183 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584533103.155046:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584533103.155047:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584533103.155049:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584533103.155055:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533103.155057:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533103.155057:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533103.155058:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533103.155059:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533103.155060:0:4532:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584533103.155061:0:4532:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584533103.155063:0:4532:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14902 00000100:00080000:2.0:1584533103.155065:0:4532:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584533103.155075:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533103.155076:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533103.155076:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533103.155077:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533103.155079:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533103.155080:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533103.155081:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533103.155082:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533103.210518:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533103.210524:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533103.210526:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533103.210527:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533103.210529:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533103.210531:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533103.210532:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533103.210534:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533183 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584533103.210544:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078398432 new x1661492078398496 00000800:00020000:2.0:1584533108.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533108.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533108.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533108.916526:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533108.928611:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533103/real 1584533103] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533190 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533108.928620:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533108.928621:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533108.928623:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533108.928630:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533108.928631:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533108.928633:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533108.928635:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533108.928635:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533108.928637:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533108.928637:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533108.928639:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14909 00000100:00080000:8.0:1584533108.928641:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533108.928654:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533108.928655:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533108.928656:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533108.928657:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533108.928658:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533108.928659:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533108.928660:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533108.928661:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533108.956963:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533108.956969:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533108.956970:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533108.956971:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533108.956973:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533108.956975:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533108.956977:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533108.956979:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533190 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533108.957007:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078398496 new x1661492078398528 00000800:00020000:4.0:1584533114.916058:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533114.916065:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533114.916069:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533114.916623:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533114.928711:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533108/real 1584533108] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533195 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533114.928731:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533114.928733:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533114.928734:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533114.928752:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533114.928755:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533114.928758:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533114.928759:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533114.928760:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533114.928762:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533114.928762:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533114.928764:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14914 00000100:00080000:8.0:1584533114.928767:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533114.928775:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533114.928776:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533114.928777:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533114.928778:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533114.928779:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533114.928780:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533114.928781:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533114.928782:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533114.979297:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533114.979305:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533114.979307:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533114.979308:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533114.979310:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533114.979312:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533114.979314:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533114.979316:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533195 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533114.979326:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078398528 new x1661492078398608 00000800:00020000:4.0:1584533120.916084:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533120.916086:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533120.916088:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533120.916524:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533120.928603:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533114/real 1584533114] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533201 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533120.928612:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533120.928613:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533120.928616:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533120.928622:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533120.928624:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533120.928625:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533120.928626:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533120.928627:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533120.928628:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533120.928629:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533120.928631:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14920 00000100:00080000:8.0:1584533120.928633:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533120.928643:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533120.928644:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533120.928645:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533120.928646:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533120.928647:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533120.928648:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533120.928649:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533120.928650:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533120.986800:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533120.986806:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533120.986808:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533120.986809:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533120.986810:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533120.986813:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533120.986814:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533120.986816:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533201 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533120.986845:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078398608 new x1661492078398656 00000800:00000100:0.0:1584533127.140463:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533127.140928:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533127.153000:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533120/real 1584533120] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533207 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533127.153012:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533127.153013:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533127.153014:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533127.153020:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533127.153022:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533127.153023:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533127.153024:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533127.153025:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533127.153039:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533127.153040:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533127.153042:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14926 00000100:00080000:8.0:1584533127.153044:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533127.153054:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533127.153055:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533127.153056:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533127.153057:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533127.153058:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533127.153059:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533127.153070:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533127.153071:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533127.213897:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533127.213903:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533127.213905:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533127.213906:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533127.213908:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533127.213910:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533127.213911:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533127.213913:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533207 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533127.213943:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078398656 new x1661492078398688 00000800:00020000:2.0:1584533132.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533132.916081:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533132.916083:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533132.916516:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533132.928593:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533127/real 1584533127] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533214 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533132.928601:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533132.928602:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533132.928603:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533132.928609:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533132.928611:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533132.928612:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533132.928613:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533132.928614:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533132.928615:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533132.928616:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533132.928618:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14933 00000100:00080000:8.0:1584533132.928620:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533132.928629:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533132.928630:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533132.928631:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533132.928632:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533132.928633:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533132.928634:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533132.928636:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533132.928637:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533132.984833:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533132.984839:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533132.984840:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533132.984841:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533132.984842:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533132.984844:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533132.984846:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533132.984848:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533214 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533132.984857:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078398688 new x1661492078398736 00000800:00020000:2.0:1584533138.916099:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533138.916105:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533138.916109:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533138.916658:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533138.928747:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533132/real 1584533132] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533219 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533138.928765:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533138.928767:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533138.928768:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533138.928782:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533138.928784:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533138.928787:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533138.928788:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533138.928788:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533138.928801:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533138.928801:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533138.928803:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14938 00000100:00080000:8.0:1584533138.928805:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533138.928810:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533138.928811:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533138.928812:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533138.928813:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533138.928814:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533138.928816:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533138.928816:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533138.928817:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533138.975198:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533138.975207:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533138.975210:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533138.975212:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533138.975215:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533138.975219:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533138.975222:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533138.975226:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533219 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533138.975247:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078398736 new x1661492078398800 00000800:00020000:2.0:1584533144.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533144.916081:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533144.916083:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533144.916522:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533144.928599:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533138/real 1584533138] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533225 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533144.928607:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533144.928608:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533144.928609:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533144.928615:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533144.928617:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533144.928617:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533144.928618:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533144.928619:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533144.928621:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533144.928621:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533144.928623:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14944 00000100:00080000:8.0:1584533144.928625:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533144.928633:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533144.928634:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533144.928636:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533144.928637:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533144.928638:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533144.928639:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533144.928640:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533144.928641:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533144.993524:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533144.993528:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533144.993529:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533144.993530:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533144.993532:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533144.993533:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533144.993535:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533144.993537:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533225 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533144.993565:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078398800 new x1661492078398832 00000800:00000100:0.0:1584533151.141301:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533151.141713:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533151.153799:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533144/real 1584533144] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533231 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533151.153806:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533151.153807:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533151.153809:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533151.153814:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533151.153816:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533151.153817:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533151.153818:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533151.153819:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533151.153820:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533151.153821:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533151.153822:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14950 00000100:00080000:8.0:1584533151.153824:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533151.153833:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533151.153834:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533151.153835:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533151.153836:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533151.153836:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533151.153837:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533151.153838:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533151.153839:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533151.191639:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533151.191644:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533151.191646:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533151.191646:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533151.191648:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533151.191650:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533151.191651:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533151.191653:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533231 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533151.191704:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078398832 new x1661492078398880 00000800:00020000:2.0:1584533156.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533156.938890:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533156.963066:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533156.963475:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533156.975547:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533151/real 1584533151] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533238 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533156.975555:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533156.975556:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533156.975557:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533156.975563:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533156.975565:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533156.975565:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533156.975567:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533156.975567:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533156.975569:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533156.975569:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533156.975572:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14957 00000100:00080000:8.0:1584533156.975574:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533156.975582:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533156.975583:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533156.975583:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533156.975585:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533156.975585:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533156.975587:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533156.975587:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533156.975588:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533157.004277:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533157.004281:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533157.004282:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533157.004283:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533157.004284:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533157.004286:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533157.004287:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533157.004289:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533238 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533157.004318:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078398880 new x1661492078398912 00000800:00020000:4.0:1584533162.916083:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533162.916086:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533162.916089:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533162.916568:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533162.928682:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533157/real 1584533157] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533244 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533162.928693:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533162.928695:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533162.928697:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533162.928715:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533162.928717:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533162.928723:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533162.928724:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533162.928725:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533162.928728:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533162.928729:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533162.928731:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14962 00000100:00080000:8.0:1584533162.928734:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533162.928742:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533162.928743:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533162.928744:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533162.928745:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533162.928746:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533162.928747:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533162.928748:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533162.928749:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533162.991199:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533162.991206:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533162.991208:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533162.991209:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533162.991211:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533162.991213:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533162.991215:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533162.991217:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533244 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533162.991246:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078398912 new x1661492078398992 00000800:00020000:2.0:1584533168.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533168.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533168.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533168.916505:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533168.928563:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533162/real 1584533162] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533249 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533168.928573:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533168.928575:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533168.928576:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533168.928583:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533168.928585:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533168.928586:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533168.928588:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533168.928589:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533168.928591:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533168.928592:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533168.928594:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14968 00000100:00080000:8.0:1584533168.928596:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533168.928612:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533168.928613:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533168.928614:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533168.928615:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533168.928616:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533168.928617:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533168.928618:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533168.928619:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533168.975741:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533168.975748:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533168.975750:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533168.975751:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533168.975753:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533168.975755:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533168.975757:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533168.975759:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533249 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533168.975789:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078398992 new x1661492078399024 00000800:00000100:0.0:1584533175.141602:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533175.142333:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533175.154411:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533168/real 1584533168] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533255 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533175.154422:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533175.154424:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533175.154426:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533175.154434:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533175.154437:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533175.154438:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533175.154440:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533175.154440:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533175.154442:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533175.154443:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533175.154445:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14974 00000100:00080000:8.0:1584533175.154449:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533175.154466:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533175.154467:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533175.154468:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533175.154469:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533175.154470:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533175.154471:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533175.154472:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533175.154473:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533175.160379:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533175.160387:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533175.160389:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533175.160390:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533175.160392:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533175.160394:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533175.160396:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533175.160398:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533255 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533175.160429:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078399024 new x1661492078399056 00000800:00020000:0.0:1584533180.916068:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584533180.916071:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584533180.916073:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533180.916487:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533180.928544:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533175/real 1584533175] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533262 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533180.928553:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533180.928554:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533180.928557:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533180.928563:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533180.928565:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533180.928566:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533180.928567:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533180.928568:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533180.928569:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533180.928570:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533180.928572:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14981 00000100:00080000:8.0:1584533180.928574:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533180.928584:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533180.928586:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533180.928586:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533180.928588:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533180.928588:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533180.928590:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533180.928590:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533180.928591:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533180.934393:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533180.934398:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533180.934400:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533180.934401:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533180.934403:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533180.934405:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533180.934407:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533180.934409:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533262 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533180.934439:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078399056 new x1661492078399104 00000800:00020000:2.0:1584533186.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533186.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533186.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533186.916485:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533186.928578:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533180/real 1584533180] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533267 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533186.928591:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533186.928593:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533186.928594:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533186.928609:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533186.928611:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533186.928618:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533186.928619:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533186.928620:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533186.928621:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533186.928622:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533186.928624:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14986 00000100:00080000:8.0:1584533186.928626:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533186.928633:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533186.928634:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533186.928635:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533186.928636:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533186.928637:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533186.928638:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533186.928639:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533186.928640:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533186.934437:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533186.934443:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533186.934445:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533186.934446:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533186.934448:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533186.934450:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533186.934451:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533186.934453:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533267 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533186.934482:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078399104 new x1661492078399168 00000800:00020000:2.0:1584533192.916053:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533192.916057:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533192.916059:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533192.916495:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533192.928546:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533186/real 1584533186] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533273 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533192.928555:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533192.928556:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533192.928557:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533192.928563:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533192.928564:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533192.928565:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533192.928566:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533192.928567:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533192.928568:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533192.928569:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533192.928571:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14992 00000100:00080000:8.0:1584533192.928573:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533192.928582:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533192.928583:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533192.928584:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533192.928585:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533192.928586:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533192.928587:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533192.928589:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533192.928590:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533192.934462:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533192.934467:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533192.934469:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533192.934470:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533192.934471:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533192.934473:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533192.934475:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533192.934477:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533273 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533192.934507:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078399168 new x1661492078399216 00000800:00000100:0.0:1584533199.141188:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533199.141711:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533199.153763:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533192/real 1584533192] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533199.153771:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533199.153772:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533199.153773:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533199.153778:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533199.153780:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533199.153781:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533199.153782:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533199.153783:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533199.153784:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533199.153784:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533199.153786:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14998 00000100:00080000:8.0:1584533199.153788:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533199.153797:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533199.153798:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533199.153799:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533199.153800:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533199.153802:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533199.153803:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533199.153804:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533199.153805:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533199.200928:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533199.200932:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533199.200934:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533199.200934:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533199.200936:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533199.200937:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533199.200939:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533199.200941:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533199.200972:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078399216 new x1661492078399248 00000800:00020000:2.0:1584533204.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533204.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533204.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533204.916470:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533204.928522:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533199/real 1584533199] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533204.928530:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533204.928531:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533204.928532:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533204.928538:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533204.928539:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533204.928540:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533204.928541:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533204.928542:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533204.928543:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533204.928544:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533204.928545:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15005 00000100:00080000:8.0:1584533204.928547:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533204.928556:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533204.928557:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533204.928559:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533204.928560:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533204.928560:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533204.928562:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533204.928562:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533204.928563:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533204.948801:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533204.948806:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533204.948807:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533204.948808:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533204.948809:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533204.948811:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533204.948812:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533204.948814:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533204.948843:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078399248 new x1661492078399280 00000800:00020000:2.0:1584533210.916053:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533210.916056:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533210.916058:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533210.916469:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533210.928523:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533204/real 1584533204] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533210.928532:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533210.928533:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533210.928534:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533210.928549:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533210.928550:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533210.928556:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533210.928558:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533210.928558:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533210.928560:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533210.928560:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533210.928562:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15010 00000100:00080000:8.0:1584533210.928564:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533210.928571:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533210.928572:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533210.928573:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533210.928574:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533210.928574:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533210.928576:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533210.928576:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533210.928577:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533210.947449:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533210.947454:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533210.947456:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533210.947457:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533210.947458:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533210.947460:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533210.947462:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533210.947464:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584533210.947494:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078399280 new x1661492078399360 00000800:00020000:4.0:1584533216.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533216.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533216.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533216.916517:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584533216.928575:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533210/real 1584533210] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584533216.928583:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584533216.928584:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584533216.928585:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584533216.928591:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533216.928593:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533216.928594:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533216.928595:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533216.928596:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533216.928597:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584533216.928598:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533216.928599:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15016 00000100:00080000:10.0:1584533216.928601:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584533216.928610:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533216.928612:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533216.928612:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533216.928614:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533216.928614:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533216.928615:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533216.928616:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533216.928618:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533216.974819:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533216.974823:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533216.974824:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533216.974825:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533216.974827:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533216.974828:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533216.974830:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533216.974832:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584533216.974861:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078399360 new x1661492078399392 00000800:00000100:0.0:1584533223.142211:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533223.142630:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584533223.154680:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533216/real 1584533216] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584533223.154688:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584533223.154690:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584533223.154691:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584533223.154697:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533223.154699:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533223.154700:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533223.154701:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533223.154702:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533223.154703:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584533223.154704:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533223.154706:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15022 00000100:00080000:10.0:1584533223.154708:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584533223.154717:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533223.154718:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533223.154719:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533223.154720:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533223.154721:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533223.154723:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533223.154723:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533223.154725:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533223.201792:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533223.201798:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533223.201799:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533223.201800:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533223.201802:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533223.201804:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533223.201805:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533223.201807:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584533223.201816:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078399392 new x1661492078399440 00000800:00000100:0.0:1584533233.142360:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533233.142782:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584533233.154837:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533223/real 1584533223] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584533233.154848:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584533233.154850:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584533233.154852:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584533233.154875:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533233.154877:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533233.154884:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533233.154885:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533233.154886:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533233.154888:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584533233.154889:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533233.154891:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15029 00000100:00080000:10.0:1584533233.154895:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584533233.154902:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533233.154904:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533233.154905:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533233.154907:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533233.154908:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533233.154909:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533233.154910:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533233.154911:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533233.204313:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533233.204321:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533233.204323:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533233.204324:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533233.204326:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533233.204328:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533233.204330:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533233.204332:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584533233.204341:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078399440 new x1661492078399504 00000800:00020000:2.0:1584533238.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533238.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533238.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533238.916514:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584533238.928619:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533233/real 1584533233] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533320 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584533238.928628:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584533238.928630:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584533238.928631:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584533238.928639:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533238.928641:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533238.928642:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533238.928643:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533238.928644:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533238.928645:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584533238.928646:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533238.928648:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15039 00000100:00080000:10.0:1584533238.928651:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584533238.928662:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533238.928665:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533238.928665:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533238.928667:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533238.928668:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533238.928669:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533238.928670:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533238.928671:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533238.988337:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533238.988344:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533238.988346:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533238.988347:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533238.988348:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533238.988351:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533238.988352:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533238.988354:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533320 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584533238.988387:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078399504 new x1661492078399552 00000800:00020000:2.0:1584533244.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533244.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 6, oc: 0, rc: 8 00000800:00000100:2.0:1584533244.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533244.916458:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584533244.928512:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533238/real 1584533238] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533325 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584533244.928521:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584533244.928523:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584533244.928524:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584533244.928530:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533244.928532:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533244.928533:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533244.928534:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533244.928535:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533244.928537:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584533244.928537:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533244.928539:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15044 00000100:00080000:10.0:1584533244.928542:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584533244.928552:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533244.928554:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533244.928555:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533244.928556:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533244.928557:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533244.928558:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533244.928558:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533244.928560:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533244.975540:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533244.975547:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533244.975548:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533244.975549:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533244.975552:0:25724:0:(client.c:2816:ptlrpc_resend_req()) @@@ going to resend req@ffff8d2babe44c80 x1661492078399568/t0(0) o17->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 456/432 e 0 to 0 dl 1584533298 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000100:00080000:12.0:1584533244.975558:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533244.975560:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533244.975561:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533244.975563:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533325 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584533244.975571:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078399552 new x1661492078399616 00000800:00020000:2.0:1584533250.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533250.916085:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533250.916089:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533250.916565:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584533250.928654:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533244/real 1584533244] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533331 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584533250.928661:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584533250.928674:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584533250.928675:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584533250.928681:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533250.928682:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533250.928683:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533250.928684:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533250.928685:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533250.928686:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584533250.928688:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533250.928689:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15050 00000100:00080000:10.0:1584533250.928691:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584533250.928700:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533250.928701:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533250.928702:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533250.928703:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533250.928704:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533250.928705:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533250.928705:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533250.928706:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533250.977100:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533250.977104:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533250.977106:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533250.977107:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533250.977108:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533250.977110:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533250.977111:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533250.977113:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533331 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584533250.977123:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078399616 new x1661492078399648 00000800:00000100:0.0:1584533257.140919:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533257.141337:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584533257.153390:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533250/real 1584533250] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533337 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584533257.153398:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584533257.153399:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584533257.153400:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584533257.153416:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533257.153417:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533257.153418:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533257.153419:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533257.153421:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533257.153422:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584533257.153423:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533257.153424:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15056 00000100:00080000:10.0:1584533257.153426:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584533257.153432:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533257.153433:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533257.153434:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533257.153435:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533257.153435:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533257.153437:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533257.153437:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533257.153438:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533257.200977:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533257.200981:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533257.200983:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533257.200984:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533257.200985:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533257.200988:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533257.200989:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533257.200991:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533337 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584533257.200999:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078399648 new x1661492078399712 00000800:00020000:2.0:1584533262.916073:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533262.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533262.916083:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533262.916503:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584533262.928593:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533257/real 1584533257] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533344 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584533262.928601:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584533262.928602:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584533262.928603:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584533262.928619:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533262.928621:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533262.928629:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533262.928630:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533262.928630:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533262.928632:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584533262.928632:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533262.928634:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15063 00000100:00080000:10.0:1584533262.928647:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584533262.928652:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533262.928653:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533262.928654:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533262.928655:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533262.928656:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533262.928657:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533262.928657:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533262.928659:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533262.974886:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533262.974890:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533262.974891:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533262.974892:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533262.974894:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533262.974895:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533262.974897:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533262.974899:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533344 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584533262.974928:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078399712 new x1661492078399760 00000800:00020000:2.0:1584533268.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533268.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533268.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533268.916524:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584533268.928585:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533262/real 1584533262] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533349 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584533268.928593:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584533268.928594:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584533268.928596:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584533268.928604:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533268.928605:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533268.928606:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533268.928607:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533268.928608:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533268.928610:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584533268.928610:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533268.928612:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15068 00000100:00080000:10.0:1584533268.928614:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584533268.928623:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533268.928625:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533268.928625:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533268.928626:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533268.928627:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533268.928628:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533268.928629:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533268.928630:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533268.980299:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533268.980304:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533268.980305:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533268.980306:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533268.980308:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533268.980310:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533268.980311:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533268.980313:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533349 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584533268.980343:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078399760 new x1661492078399808 00000800:00020000:2.0:1584533274.916102:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533274.916108:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533274.916112:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533274.916545:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584533274.928629:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533268/real 1584533268] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533355 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584533274.928648:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584533274.928650:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584533274.928651:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584533274.928656:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533274.928658:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533274.928659:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533274.928660:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533274.928661:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533274.928662:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584533274.928663:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533274.928664:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15074 00000100:00080000:10.0:1584533274.928666:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584533274.928674:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533274.928675:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533274.928676:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533274.928677:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533274.928678:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533274.928679:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533274.928680:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533274.928681:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533274.980172:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533274.980176:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533274.980177:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533274.980178:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533274.980179:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533274.980181:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533274.980182:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533274.980184:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533355 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584533274.980215:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078399808 new x1661492078399840 00000800:00000100:0.0:1584533281.141856:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533281.142297:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584533281.154347:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533274/real 1584533274] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533361 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584533281.154356:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584533281.154357:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584533281.154358:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584533281.154372:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533281.154374:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533281.154375:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533281.154376:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533281.154376:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533281.154378:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584533281.154378:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533281.154380:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15080 00000100:00080000:10.0:1584533281.154382:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584533281.154387:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533281.154388:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533281.154389:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533281.154390:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533281.154391:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533281.154392:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533281.154392:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533281.154394:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533281.175088:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533281.175092:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533281.175093:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533281.175094:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533281.175096:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533281.175098:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533281.175099:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533281.175101:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533361 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584533281.175109:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078399840 new x1661492078399888 00000800:00020000:2.0:1584533286.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533286.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533286.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533286.916578:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584533286.928695:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533281/real 1584533281] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533368 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584533286.928704:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584533286.928706:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584533286.928708:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584533286.928715:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533286.928717:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533286.928729:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533286.928730:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533286.928731:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533286.928733:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584533286.928734:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533286.928736:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15087 00000100:00080000:10.0:1584533286.928738:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584533286.928745:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533286.928746:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533286.928747:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533286.928748:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533286.928748:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533286.928750:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533286.928751:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533286.928753:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533286.976823:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533286.976828:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533286.976830:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533286.976831:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533286.976833:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533286.976835:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533286.976837:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533286.976839:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533368 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584533286.976867:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078399888 new x1661492078399952 00000800:00020000:2.0:1584533292.916066:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533292.916070:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533292.916072:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533292.916582:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584533292.928718:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533286/real 1584533286] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533373 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584533292.928730:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584533292.928731:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584533292.928733:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584533292.928741:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533292.928743:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533292.928744:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533292.928745:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533292.928746:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533292.928748:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584533292.928749:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533292.928751:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15092 00000100:00080000:10.0:1584533292.928755:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584533292.928771:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533292.928773:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533292.928773:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533292.928775:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533292.928776:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533292.928778:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533292.928779:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533292.928780:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533292.963429:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533292.963436:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533292.963438:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533292.963439:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533292.963441:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533292.963443:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533292.963445:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533292.963446:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533373 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584533292.963481:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078399952 new x1661492078399984 00000800:00020000:10.0:1584533298.916150:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:10.0:1584533298.916155:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:10.0:1584533298.916160:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533298.916635:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584533298.928722:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533292/real 1584533292] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533379 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584533298.928743:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584533298.928744:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584533298.928746:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584533298.928753:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533298.928755:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533298.928756:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533298.928757:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533298.928758:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533298.928760:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584533298.928760:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533298.928762:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15098 00000100:00080000:10.0:1584533298.928765:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584533298.928777:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533298.928779:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533298.928780:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533298.928781:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533298.928782:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533298.928783:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533298.928784:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533298.928785:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533298.974696:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533298.974702:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533298.974704:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533298.974705:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533298.974707:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533298.974710:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533298.974711:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533298.974713:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533379 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584533298.974743:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078399984 new x1661492078400032 00000100:00000400:18.0:1584533299.975222:0:25731:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584533239/real 1584533239] req@ffff8d2babe46780 x1661492078399584/t0(0) o17->public1-OST0001-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 456/432 e 0 to 1 dl 1584533299 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 00000100:02000400:18.0:1584533299.975231:0:25731:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection to public1-OST0001 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584533299.975232:0:25731:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584533299.975234:0:25731:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0001_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584533299.975241:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533299.975243:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533299.975244:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533299.975245:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:18.0:1584533299.975246:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584533299.975248:0:4066:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 12752 00000100:00080000:18.0:1584533299.975250:0:4066:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584533299.975260:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533299.975262:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533299.975263:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533299.975264:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533299.975264:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533299.975266:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533299.975266:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533299.975267:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533299.975268:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533299.975269:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000800:00000100:0.0:1584533305.140637:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533305.141041:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584533305.153082:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533298/real 1584533298] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533385 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584533305.153089:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584533305.153091:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584533305.153092:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584533305.153106:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533305.153108:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533305.153109:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533305.153110:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533305.153110:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533305.153112:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584533305.153112:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533305.153115:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15104 00000100:00080000:10.0:1584533305.153117:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584533305.153123:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533305.153124:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533305.153125:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533305.153126:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533305.153127:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533305.153128:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533305.153129:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533305.153130:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533305.200290:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533305.200296:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533305.200298:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533305.200299:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533305.200302:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533305.200305:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533305.200306:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533305.200308:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533385 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584533305.200317:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078400032 new x1661492078400096 00000100:00080000:18.0:1584533306.153060:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:18.0:1584533306.153062:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0001_UUID on 10.10.2.22@o2ib failed (-110) 00000800:00020000:4.0:1584533310.916098:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533310.916100:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533310.916102:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533310.916507:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584533310.928582:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533305/real 1584533305] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533392 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584533310.928589:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584533310.928601:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584533310.928603:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584533310.928609:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533310.928610:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533310.928611:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533310.928612:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533310.928613:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533310.928615:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584533310.928616:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533310.928617:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15111 00000100:00080000:10.0:1584533310.928620:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584533310.928628:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533310.928629:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533310.928630:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533310.928631:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533310.928631:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533310.928633:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533310.928633:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533310.928635:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533310.981612:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533310.981616:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533310.981617:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533310.981618:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533310.981619:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533310.981621:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533310.981623:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533310.981624:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533392 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584533310.981670:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078400096 new x1661492078400128 00000800:00020000:2.0:1584533316.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533316.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533316.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533316.916504:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584533316.928561:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533310/real 1584533310] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533397 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584533316.928569:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584533316.928570:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584533316.928572:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584533316.928577:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533316.928579:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533316.928580:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533316.928582:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533316.928582:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533316.928584:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584533316.928585:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533316.928586:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15116 00000100:00080000:10.0:1584533316.928588:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584533316.928597:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533316.928598:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533316.928599:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533316.928600:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533316.928601:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533316.928602:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533316.928603:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533316.928604:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533316.980793:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533316.980797:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533316.980799:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533316.980799:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533316.980801:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533316.980803:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533316.980804:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533316.980806:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533397 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584533316.980835:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078400128 new x1661492078400176 00000800:00020000:2.0:1584533322.916069:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533322.916071:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533322.916073:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533322.916508:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584533322.928599:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533316/real 1584533316] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533403 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584533322.928606:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584533322.928618:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584533322.928619:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584533322.928625:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533322.928639:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533322.928640:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:10.0:1584533322.928641:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:10.0:1584533322.928642:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533322.928643:0:334:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15105 00000100:00080000:10.0:1584533322.928645:0:334:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 12728 00000100:00080000:10.0:1584533322.928647:0:334:0:(import.c:596:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0001 (at 10.10.2.21@o2ib) 00000100:00080000:10.0:1584533322.928648:0:334:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.21@o2ib/10.10.2.21@o2ib 00000100:00080000:10.0:1584533322.928657:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533322.928658:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533322.928659:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533322.928660:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584533322.928660:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533322.928662:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15122 00000100:00080000:10.0:1584533322.928663:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584533322.928666:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533322.928667:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533322.928668:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533322.928669:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533322.928670:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533322.928671:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533322.928672:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533322.928673:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533322.928887:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:18.0:1584533322.928890:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0001_UUID on 10.10.2.21@o2ib failed (-19) 00000100:00080000:18.0:1584533322.982167:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533322.982171:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533322.982172:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533322.982173:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533322.982174:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533322.982176:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533322.982178:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533322.982179:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533403 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584533322.982212:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078400176 new x1661492078400224 00000800:00000100:0.0:1584533329.141464:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533329.141935:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584533329.153991:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533322/real 1584533322] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533409 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584533329.154000:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584533329.154001:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584533329.154003:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584533329.154017:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533329.154019:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533329.154020:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533329.154021:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533329.154022:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533329.154023:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584533329.154024:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533329.154025:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15128 00000100:00080000:10.0:1584533329.154044:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584533329.154051:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533329.154052:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533329.154052:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533329.154054:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533329.154055:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533329.154057:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533329.154057:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533329.154058:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533329.216037:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533329.216042:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533329.216043:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533329.216044:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533329.216045:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533329.216048:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533329.216049:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533329.216051:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533409 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584533329.216080:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078400224 new x1661492078400288 00000800:00020000:2.0:1584533334.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533334.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533334.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533334.916515:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584533334.928603:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533329/real 1584533329] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533416 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584533334.928623:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584533334.928624:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584533334.928625:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584533334.928630:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533334.928632:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533334.928633:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533334.928634:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533334.928635:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533334.928636:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584533334.928637:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533334.928638:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15135 00000100:00080000:10.0:1584533334.928640:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584533334.928648:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533334.928649:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533334.928651:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533334.928652:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533334.928653:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533334.928654:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533334.928654:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533334.928655:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533334.980922:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533334.980925:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533334.980927:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533334.980928:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533334.980929:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533334.980931:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533334.980932:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533334.980934:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533416 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584533334.980965:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078400288 new x1661492078400320 00000800:00020000:2.0:1584533340.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533340.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533340.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533340.916527:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584533340.928582:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533334/real 1584533334] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533421 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584533340.928591:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584533340.928592:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584533340.928593:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584533340.928599:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533340.928600:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533340.928601:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533340.928602:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533340.928603:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533340.928604:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584533340.928605:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533340.928607:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15140 00000100:00080000:10.0:1584533340.928609:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584533340.928618:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533340.928619:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533340.928620:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533340.928621:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533340.928621:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533340.928623:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533340.928623:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533340.928624:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533340.992434:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533340.992438:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533340.992440:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533340.992440:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533340.992442:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533340.992444:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533340.992445:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533340.992447:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533421 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584533340.992476:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078400320 new x1661492078400352 00000800:00020000:8.0:1584533346.916111:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584533346.916113:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584533346.916115:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533346.916544:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584533346.928630:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533340/real 1584533340] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533427 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584533346.928650:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584533346.928651:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584533346.928652:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584533346.928658:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533346.928660:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533346.928661:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:2.0:1584533346.928662:0:4532:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:2.0:1584533346.928663:0:4532:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584533346.928665:0:4532:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15105 00000100:00080000:2.0:1584533346.928666:0:4532:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 15128 00000100:00080000:2.0:1584533346.928668:0:4532:0:(import.c:570:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: tried all connections, increasing latency to 7s 00000100:00080000:2.0:1584533346.928670:0:4532:0:(import.c:596:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0001 (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584533346.928672:0:4532:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584533346.928681:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533346.928682:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533346.928683:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533346.928684:0:4532:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584533346.928685:0:4532:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584533346.928686:0:4532:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15146 00000100:00080000:2.0:1584533346.928687:0:4532:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584533346.928701:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533346.928702:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533346.928703:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533346.928704:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533346.928705:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533346.928706:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533346.928707:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533346.928708:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533346.993691:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533346.993700:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533346.993703:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0001_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533346.993705:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533346.993708:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533346.993712:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533346.993716:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533346.993720:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727c80): req@ffff8d2babe46780 x1661492078399584/t0(0) o17->public1-OST0001-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 456/432 e 0 to 1 dl 1584533299 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 00000100:00080000:18.0:1584533346.993738:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533346.993742:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533346.993743:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533346.993744:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533346.993746:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533346.993749:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533346.993752:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533346.993755:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533427 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584533346.993773:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078400352 new x1661492078400416 00000800:00000100:0.0:1584533353.142347:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533353.142780:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584533353.154859:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533346/real 1584533346] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533433 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584533353.154875:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584533353.154877:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584533353.154879:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584533353.154903:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533353.154905:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533353.154906:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533353.154907:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533353.154908:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533353.154910:0:4532:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584533353.154911:0:4532:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584533353.154913:0:4532:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15152 00000100:00080000:2.0:1584533353.154916:0:4532:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584533353.154924:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533353.154925:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533353.154926:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533353.154927:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533353.154928:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533353.154929:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533353.154930:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533353.154931:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533353.207531:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533353.207539:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533353.207541:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533353.207542:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533353.207544:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533353.207547:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533353.207548:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533353.207550:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533433 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584533353.207580:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078400416 new x1661492078400464 00000800:00020000:8.0:1584533358.916057:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584533358.916060:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584533358.916063:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533358.916578:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584533358.928669:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533353/real 1584533353] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533440 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584533358.928689:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584533358.928691:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584533358.928693:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584533358.928700:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533358.928702:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533358.928703:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533358.928705:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533358.928706:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533358.928707:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584533358.928708:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584533358.928710:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15159 00000100:00080000:4.0:1584533358.928713:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584533358.928727:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533358.928728:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533358.928729:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533358.928730:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533358.928731:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533358.928733:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533358.928734:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533358.928735:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533358.977364:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533358.977370:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533358.977372:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533358.977374:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533358.977375:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533358.977378:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533358.977379:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533358.977381:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533440 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584533358.977415:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078400464 new x1661492078400512 00000800:00020000:8.0:1584533364.916082:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584533364.916085:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584533364.916088:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533364.916557:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584533364.928615:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533358/real 1584533358] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533445 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584533364.928624:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584533364.928626:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584533364.928627:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584533364.928634:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533364.928636:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533364.928637:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533364.928639:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533364.928639:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533364.928641:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584533364.928642:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584533364.928644:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15164 00000100:00080000:4.0:1584533364.928646:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584533364.928659:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533364.928660:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533364.928661:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533364.928664:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533364.928665:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533364.928666:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533364.928667:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533364.928668:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533364.994231:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533364.994238:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533364.994241:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533364.994242:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533364.994244:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533364.994247:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533364.994248:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533364.994250:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533445 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584533364.994281:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078400512 new x1661492078400544 00000800:00020000:4.0:1584533370.916107:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533370.916109:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533370.916111:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533370.916575:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584533370.928687:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533364/real 1584533364] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533451 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584533370.928695:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584533370.928696:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584533370.928697:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584533370.928703:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533370.928704:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533370.928713:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533370.928715:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533370.928715:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533370.928717:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584533370.928717:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584533370.928719:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15170 00000100:00080000:4.0:1584533370.928722:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584533370.928728:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533370.928729:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533370.928730:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533370.928731:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533370.928731:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533370.928732:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533370.928733:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533370.928734:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533370.980987:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533370.980992:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533370.980994:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533370.980996:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533370.980997:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533370.980999:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533370.981001:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533370.981002:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533451 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533370.981012:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078400544 new x1661492078400592 00000800:00000100:0.0:1584533377.141075:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533377.141707:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533377.153787:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533370/real 1584533370] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533457 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533377.153795:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533377.153796:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533377.153797:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533377.153812:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533377.153814:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533377.153815:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533377.153816:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533377.153816:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533377.153818:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533377.153818:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533377.153820:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15176 00000100:00080000:8.0:1584533377.153822:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533377.153828:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533377.153829:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533377.153830:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533377.153831:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533377.153831:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533377.153833:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533377.153833:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533377.153834:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533377.196332:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533377.196337:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533377.196339:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533377.196340:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533377.196341:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533377.196343:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533377.196345:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533377.196346:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533457 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533377.196376:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078400592 new x1661492078400656 00000800:00020000:4.0:1584533382.916082:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533382.916085:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533382.916087:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533382.916733:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533382.928822:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533377/real 1584533377] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533464 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533382.928829:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533382.928831:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533382.928833:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533382.928838:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533382.928840:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533382.928841:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533382.928842:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533382.928843:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533382.928844:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533382.928844:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533382.928846:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15183 00000100:00080000:8.0:1584533382.928848:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533382.928856:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533382.928858:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533382.928858:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533382.928859:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533382.928860:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533382.928861:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533382.928862:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533382.928863:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533382.982584:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533382.982588:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533382.982589:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533382.982590:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533382.982591:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533382.982593:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533382.982595:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533382.982596:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533464 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533382.982631:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078400656 new x1661492078400688 00000800:00020000:8.0:1584533388.916055:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584533388.916060:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584533388.916062:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533388.916532:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533388.928609:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533382/real 1584533382] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533469 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533388.928617:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533388.928618:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533388.928619:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533388.928626:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533388.928627:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533388.928628:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533388.928629:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533388.928630:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533388.928631:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533388.928632:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533388.928633:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15188 00000100:00080000:8.0:1584533388.928635:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533388.928644:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533388.928645:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533388.928646:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533388.928647:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533388.928648:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533388.928649:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533388.928651:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533388.928652:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533388.987902:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533388.987906:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533388.987907:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533388.987908:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533388.987910:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533388.987912:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533388.987913:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533388.987915:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533469 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533388.987943:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078400688 new x1661492078400736 00000800:00020000:2.0:1584533394.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533394.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533394.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533394.916497:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533394.928574:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533388/real 1584533388] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533475 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533394.928582:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533394.928583:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533394.928584:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533394.928590:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533394.928591:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533394.928599:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533394.928601:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533394.928601:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533394.928603:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533394.928603:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533394.928605:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15194 00000100:00080000:8.0:1584533394.928607:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533394.928612:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533394.928613:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533394.928614:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533394.928615:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533394.928616:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533394.928618:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533394.928618:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533394.928619:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533394.978061:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533394.978065:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533394.978066:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533394.978067:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533394.978069:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533394.978071:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533394.978072:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533394.978074:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533475 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533394.978103:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078400736 new x1661492078400784 00000800:00000100:0.0:1584533401.141798:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533401.142394:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533401.154472:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533394/real 1584533394] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533481 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533401.154479:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533401.154481:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533401.154482:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533401.154495:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533401.154497:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533401.154498:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533401.154499:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533401.154499:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533401.154501:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533401.154501:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533401.154503:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15200 00000100:00080000:8.0:1584533401.154505:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533401.154510:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533401.154512:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533401.154513:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533401.154514:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533401.154515:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533401.154516:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533401.154517:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533401.154518:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533401.211068:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533401.211072:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533401.211074:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533401.211075:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533401.211076:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533401.211078:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533401.211079:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533401.211081:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533481 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533401.211111:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078400784 new x1661492078400832 00000800:00020000:8.0:1584533406.916101:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584533406.916105:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584533406.916107:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533406.916559:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533406.928650:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533401/real 1584533401] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533488 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533406.928673:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533406.928675:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533406.928677:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533406.928686:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533406.928688:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533406.928689:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533406.928690:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533406.928691:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533406.928692:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533406.928693:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533406.928697:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15207 00000100:00080000:8.0:1584533406.928700:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533406.928719:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533406.928720:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533406.928721:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533406.928722:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533406.928723:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533406.928724:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533406.928725:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533406.928737:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533406.976915:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533406.976922:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533406.976925:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533406.976926:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533406.976928:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533406.976931:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533406.976932:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533406.976934:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533488 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533406.976968:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078400832 new x1661492078400880 00000800:00020000:2.0:1584533412.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533412.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533412.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533412.916468:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533412.928562:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533406/real 1584533406] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533493 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533412.928574:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533412.928575:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533412.928577:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533412.928586:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533412.928588:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533412.928589:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533412.928590:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533412.928591:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533412.928594:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533412.928595:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533412.928597:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15212 00000100:00080000:8.0:1584533412.928601:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533412.928618:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533412.928620:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533412.928621:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533412.928622:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533412.928622:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533412.928624:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533412.928625:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533412.928626:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533412.987149:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533412.987156:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533412.987158:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533412.987159:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533412.987162:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533412.987164:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533412.987166:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533412.987168:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533493 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533412.987198:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078400880 new x1661492078400912 00000800:00020000:6.0:1584533418.916081:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584533418.916084:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584533418.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533418.916529:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533418.928610:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533412/real 1584533412] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533499 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533418.928619:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533418.928620:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533418.928622:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533418.928628:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533418.928630:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533418.928640:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533418.928642:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533418.928643:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533418.928645:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533418.928646:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533418.928648:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15218 00000100:00080000:8.0:1584533418.928650:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533418.928657:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533418.928659:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533418.928659:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533418.928661:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533418.928661:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533418.928663:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533418.928663:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533418.928665:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533418.977244:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533418.977250:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533418.977252:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533418.977253:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533418.977255:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533418.977257:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533418.977259:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533418.977261:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533499 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533418.977290:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078400912 new x1661492078400976 00000800:00000100:0.0:1584533425.141170:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533425.141634:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533425.153702:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533418/real 1584533418] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533505 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533425.153710:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533425.153711:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533425.153712:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533425.153726:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533425.153728:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533425.153729:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533425.153730:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533425.153731:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533425.153732:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533425.153733:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533425.153734:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15224 00000100:00080000:8.0:1584533425.153736:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533425.153742:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533425.153743:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533425.153744:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533425.153745:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533425.153746:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533425.153747:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533425.153747:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533425.153749:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533425.208006:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533425.208010:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533425.208011:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533425.208012:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533425.208013:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533425.208015:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533425.208017:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533425.208018:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533505 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533425.208053:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078400976 new x1661492078401024 00000800:00020000:2.0:1584533430.916066:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533430.916068:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533430.916070:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533430.916484:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584533430.928566:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533425/real 1584533425] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533512 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584533430.928574:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584533430.928575:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584533430.928576:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584533430.928582:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533430.928584:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533430.928585:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533430.928586:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533430.928587:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533430.928588:0:4532:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584533430.928589:0:4532:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584533430.928590:0:4532:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15231 00000100:00080000:2.0:1584533430.928592:0:4532:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584533430.928602:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533430.928604:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533430.928604:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533430.928605:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533430.928606:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533430.928607:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533430.928608:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584533430.928609:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533430.986351:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533430.986355:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533430.986357:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533430.986358:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533430.986359:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533430.986361:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533430.986362:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533430.986364:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533512 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584533430.986373:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078401024 new x1661492078401056 00000800:00020000:2.0:1584533436.916057:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533436.916059:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533436.916061:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533436.916506:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584533436.928559:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533430/real 1584533430] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533517 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584533436.928567:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584533436.928568:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584533436.928569:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584533436.928574:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533436.928576:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533436.928577:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533436.928578:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533436.928582:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533436.928583:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584533436.928584:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584533436.928585:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15236 00000100:00080000:4.0:1584533436.928588:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584533436.928597:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533436.928598:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533436.928598:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533436.928600:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533436.928600:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533436.928602:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533436.928602:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533436.928603:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533436.970645:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533436.970649:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533436.970651:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533436.970651:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533436.970653:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533436.970655:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533436.970656:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533436.970658:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533517 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584533436.970667:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078401056 new x1661492078401104 00000800:00020000:2.0:1584533442.916059:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533442.916062:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533442.916063:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533442.916510:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584533442.928554:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533436/real 1584533436] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533523 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584533442.928561:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584533442.928562:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584533442.928564:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584533442.928568:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533442.928570:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533442.928580:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533442.928582:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533442.928582:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533442.928584:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584533442.928584:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584533442.928586:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15242 00000100:00080000:4.0:1584533442.928588:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584533442.928593:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533442.928595:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533442.928595:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533442.928596:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533442.928597:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533442.928598:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533442.928599:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533442.928600:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533442.990651:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533442.990655:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533442.990656:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533442.990657:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533442.990659:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533442.990661:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533442.990662:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533442.990664:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533523 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584533442.990673:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078401104 new x1661492078401152 00000800:00000100:0.0:1584533449.141925:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533449.142312:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584533449.154382:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533442/real 1584533442] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533529 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584533449.154390:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584533449.154391:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584533449.154393:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584533449.154409:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533449.154411:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533449.154412:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533449.154413:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533449.154414:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533449.154415:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584533449.154416:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584533449.154418:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15248 00000100:00080000:4.0:1584533449.154420:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584533449.154427:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533449.154428:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533449.154429:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533449.154430:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533449.154431:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533449.154432:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533449.154433:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533449.154434:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533449.207398:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533449.207403:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533449.207405:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533449.207406:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533449.207407:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533449.207409:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533449.207411:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533449.207413:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533529 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584533449.207441:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078401152 new x1661492078401216 00000800:00020000:2.0:1584533454.916055:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533454.916058:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533454.916060:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533454.916480:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584533454.928532:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533449/real 1584533449] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533536 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584533454.928540:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584533454.928542:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584533454.928543:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584533454.928548:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533454.928549:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533454.928550:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533454.928551:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533454.928552:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533454.928553:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584533454.928554:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584533454.928555:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15255 00000100:00080000:4.0:1584533454.928557:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584533454.928565:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533454.928567:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533454.928567:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533454.928568:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533454.928569:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533454.928570:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533454.928571:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533454.928572:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533454.988508:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533454.988512:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533454.988514:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533454.988514:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533454.988516:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533454.988517:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533454.988519:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533454.988521:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533536 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584533454.988531:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078401216 new x1661492078401248 00000800:00020000:0.0:1584533460.916057:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584533460.916060:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584533460.916062:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533460.916472:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584533460.928548:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533454/real 1584533454] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533541 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584533460.928557:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584533460.928558:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584533460.928559:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584533460.928565:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533460.928567:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533460.928568:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533460.928569:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533460.928570:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533460.928571:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584533460.928571:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584533460.928573:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15260 00000100:00080000:4.0:1584533460.928575:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584533460.928584:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533460.928585:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533460.928585:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533460.928587:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533460.928587:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533460.928588:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533460.928589:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584533460.928590:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533460.946483:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533460.946487:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533460.946489:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533460.946490:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533460.946491:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533460.946493:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533460.946494:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533460.946496:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533541 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584533460.946505:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078401248 new x1661492078401280 00000800:00020000:4.0:1584533466.916101:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533466.916104:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533466.916106:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533466.916498:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533466.928585:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533460/real 1584533460] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533547 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533466.928595:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533466.928597:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533466.928599:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533466.928606:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533466.928608:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533466.928621:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533466.928622:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533466.928623:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533466.928625:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533466.928626:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533466.928628:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15266 00000100:00080000:8.0:1584533466.928631:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533466.928638:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533466.928639:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533466.928640:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533466.928641:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533466.928642:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533466.928643:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533466.928644:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533466.928645:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533467.127372:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533467.127378:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533467.127380:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533467.127381:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533467.127383:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533467.127386:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533467.127387:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533467.127389:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533547 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533467.127419:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078401280 new x1661492078401344 00000800:00000100:0.0:1584533473.140708:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533473.141148:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533473.153201:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533467/real 1584533467] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533554 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533473.153217:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533473.153219:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533473.153221:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533473.153244:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533473.153246:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533473.153247:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533473.153249:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533473.153249:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533473.153251:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533473.153253:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533473.153255:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15272 00000100:00080000:8.0:1584533473.153258:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533473.153266:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533473.153267:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533473.153268:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533473.153269:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533473.153270:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533473.153271:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533473.153272:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533473.153273:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533473.189129:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533473.189136:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533473.189138:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533473.189139:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533473.189141:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533473.189144:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533473.189146:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533473.189148:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533554 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533473.189177:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078401344 new x1661492078401392 00000800:00020000:6.0:1584533478.916104:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584533478.916107:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584533478.916109:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533478.916546:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584533478.928630:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533473/real 1584533473] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533560 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584533478.928639:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584533478.928640:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584533478.928641:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584533478.928651:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533478.928653:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533478.928654:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533478.928656:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533478.928656:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533478.928658:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584533478.928659:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584533478.928660:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15279 00000100:00080000:8.0:1584533478.928663:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584533478.928672:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533478.928674:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533478.928674:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533478.928675:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533478.928676:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533478.928677:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584533478.928679:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584533478.928680:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533478.980778:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533478.980783:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533478.980785:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533478.980786:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533478.980787:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533478.980790:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533478.980791:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533478.980793:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533560 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584533478.980822:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078401392 new x1661492078401440 00000800:00020000:4.0:1584533484.916114:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533484.916116:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533484.916118:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533484.916617:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584533484.928672:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533478/real 1584533478] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533565 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584533484.928679:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584533484.928681:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584533484.928682:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584533484.928688:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533484.928689:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533484.928690:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533484.928691:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533484.928692:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533484.928693:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584533484.928694:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533484.928697:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15284 00000100:00080000:10.0:1584533484.928699:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584533484.928708:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533484.928710:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533484.928710:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533484.928711:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533484.928712:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533484.928713:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533484.928714:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533484.928715:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533484.985422:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533484.985426:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533484.985428:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533484.985428:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533484.985430:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533484.985432:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533484.985433:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533484.985435:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533565 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584533484.985443:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078401440 new x1661492078401472 00000800:00020000:2.0:1584533490.916081:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533490.916084:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533490.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533490.916462:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584533490.928554:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533484/real 1584533484] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533571 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584533490.928562:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584533490.928563:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584533490.928564:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584533490.928571:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533490.928572:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533490.928581:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533490.928583:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533490.928583:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533490.928586:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584533490.928586:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533490.928588:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15290 00000100:00080000:10.0:1584533490.928590:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584533490.928596:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533490.928597:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533490.928597:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533490.928599:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533490.928599:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533490.928600:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533490.928601:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533490.928602:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533490.979485:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533490.979489:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533490.979490:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533490.979491:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533490.979492:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533490.979494:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533490.979496:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533490.979497:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533571 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584533490.979530:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078401472 new x1661492078401520 00000800:00000100:0.0:1584533497.141276:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533497.141846:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584533497.153896:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533490/real 1584533490] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533577 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584533497.153904:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584533497.153905:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584533497.153907:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584533497.153920:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533497.153922:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533497.153923:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533497.153925:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533497.153926:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533497.153927:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584533497.153928:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584533497.153929:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15296 00000100:00080000:10.0:1584533497.153931:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584533497.153936:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533497.153938:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533497.153938:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533497.153939:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533497.153940:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533497.153941:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533497.153942:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584533497.153943:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533497.192242:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533497.192246:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533497.192248:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533497.192248:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533497.192250:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533497.192252:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533497.192253:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533497.192255:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533577 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584533497.192264:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078401520 new x1661492078401584 00000800:00020000:2.0:1584533502.916097:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533502.916103:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533502.916107:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533502.916575:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584533502.928661:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533497/real 1584533497] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533584 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584533502.928679:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584533502.928680:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584533502.928681:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584533502.928687:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584533502.928689:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533502.928689:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584533502.928691:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533502.928691:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584533502.928692:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584533502.928693:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584533502.928694:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15303 00000100:00080000:12.0:1584533502.928698:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584533502.928707:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584533502.928708:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533502.928709:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584533502.928710:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533502.928711:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584533502.928712:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533502.928712:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584533502.928713:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533502.971958:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533502.971962:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533502.971964:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533502.971965:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533502.971966:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533502.971968:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533502.971969:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533502.971971:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533584 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584533502.972004:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078401584 new x1661492078401616 00000800:00020000:2.0:1584533508.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533508.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533508.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533508.916493:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584533508.928572:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533502/real 1584533502] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533589 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584533508.928580:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584533508.928582:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584533508.928583:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584533508.928589:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584533508.928591:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533508.928591:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584533508.928593:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533508.928593:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584533508.928594:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584533508.928596:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584533508.928598:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15308 00000100:00080000:12.0:1584533508.928600:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584533508.928609:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584533508.928610:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533508.928611:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584533508.928612:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533508.928612:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584533508.928613:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533508.928614:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584533508.928615:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533508.954804:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533508.954809:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533508.954810:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533508.954811:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533508.954812:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533508.954814:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533508.954816:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533508.954817:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533589 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584533508.954849:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078401616 new x1661492078401664 00000800:00020000:2.0:1584533514.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533514.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533514.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533514.916501:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584533514.928575:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533508/real 1584533508] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533595 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584533514.928582:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584533514.928583:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584533514.928595:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584533514.928601:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584533514.928603:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533514.928610:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584533514.928611:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533514.928613:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584533514.928614:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584533514.928614:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584533514.928616:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15314 00000100:00080000:12.0:1584533514.928630:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584533514.928635:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584533514.928636:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533514.928637:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584533514.928638:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533514.928638:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584533514.928639:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533514.928640:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584533514.928641:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533514.934557:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533514.934561:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533514.934562:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533514.934563:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533514.934565:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533514.934566:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533514.934568:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533514.934569:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533595 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0F:1584533514.934599:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078401664 new x1661492078401712 00000800:00000100:0.0:1584533521.142038:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533521.142530:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533521.154579:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533514/real 1584533514] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533601 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533521.154587:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533521.154588:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533521.154590:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533521.154604:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533521.154605:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533521.154606:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533521.154607:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533521.154608:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533521.154609:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533521.154610:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533521.154612:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15320 00000100:00080000:14.0:1584533521.154614:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533521.154619:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533521.154621:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533521.154621:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533521.154622:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533521.154623:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533521.154624:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533521.154625:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533521.154627:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533521.160417:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533521.160421:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533521.160423:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533521.160423:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533521.160425:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533521.160427:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533521.160428:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533521.160430:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533601 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533521.160459:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078401712 new x1661492078401760 00000800:00020000:2.0:1584533526.916060:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533526.916063:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533526.916065:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533526.916504:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533526.928608:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533521/real 1584533521] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533608 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533526.928629:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533526.928630:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533526.928632:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533526.928640:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533526.928642:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533526.928643:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533526.928644:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533526.928645:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533526.928647:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533526.928648:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533526.928650:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15327 00000100:00080000:14.0:1584533526.928653:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533526.928667:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533526.928668:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533526.928669:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533526.928670:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533526.928671:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533526.928673:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533526.928674:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533526.928675:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533526.934815:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533526.934822:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533526.934824:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533526.934825:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533526.934827:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533526.934829:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533526.934831:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533526.934832:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533608 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533526.934867:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078401760 new x1661492078401808 00000800:00020000:4.0:1584533532.916101:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533532.916108:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533532.916112:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533532.916514:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533532.928606:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533526/real 1584533526] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533613 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533532.928627:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533532.928628:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533532.928630:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533532.928637:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533532.928639:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533532.928640:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533532.928641:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533532.928642:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533532.928643:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533532.928644:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533532.928646:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15332 00000100:00080000:14.0:1584533532.928649:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533532.928664:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533532.928666:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533532.928666:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533532.928669:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533532.928669:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533532.928670:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533532.928671:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533532.928684:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533532.934567:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533532.934574:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533532.934576:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533532.934577:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533532.934579:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533532.934582:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533532.934583:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533532.934585:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533613 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533532.934619:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078401808 new x1661492078401840 00000800:00020000:6.0:1584533538.916112:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584533538.916114:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584533538.916116:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533538.916566:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533538.928664:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533532/real 1584533532] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533619 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533538.972456:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533538.972457:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533538.972458:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533538.972462:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533538.972464:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533538.972472:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533538.972473:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533538.972474:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533538.972475:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533538.972476:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533538.972477:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15338 00000100:00080000:14.0:1584533538.972490:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533538.972495:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533538.972497:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533538.972498:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533538.972499:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533538.972500:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533538.972501:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533538.972502:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533538.972514:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533538.978232:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533538.978238:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533538.978240:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533538.978241:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533538.978242:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533538.978245:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533538.978247:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533538.978248:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533619 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533538.978278:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078401840 new x1661492078401904 00000800:00020000:4.0:1584533544.916107:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533544.916109:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533544.916111:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533544.916497:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533544.928548:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533538/real 1584533538] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533625 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533544.928555:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533544.928557:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533544.928558:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533544.928569:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533544.928571:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533544.928572:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533544.928573:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533544.928573:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533544.928575:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533544.928575:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533544.928577:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15344 00000100:00080000:14.0:1584533544.928580:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533544.928585:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533544.928587:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533544.928587:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533544.928588:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533544.928589:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533544.928590:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533544.928591:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533544.928592:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533544.934427:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533544.934431:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533544.934432:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533544.934433:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533544.934434:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533544.934436:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533544.934438:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533544.934439:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533625 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533544.934470:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078401904 new x1661492078401952 00000800:00020000:2.0:1584533550.916098:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533550.916103:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533550.916107:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533550.916587:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533550.928683:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533544/real 1584533544] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533631 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533550.928691:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533550.928692:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533550.928693:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533550.928698:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533550.928699:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533550.928700:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533550.928701:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533550.928702:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533550.928703:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533550.928705:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533550.928707:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15350 00000100:00080000:14.0:1584533550.928709:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533550.928717:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533550.928718:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533550.928719:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533550.928720:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533550.928720:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533550.928721:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533550.928722:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533550.928723:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533550.934796:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533550.934800:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533550.934802:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533550.934802:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533550.934804:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533550.934806:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533550.934807:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533550.934809:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533631 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533550.934838:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078401952 new x1661492078401984 00000800:00020000:2.0:1584533556.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533556.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533556.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533556.916497:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533556.928551:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533550/real 1584533550] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533637 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533556.928559:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533556.928560:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533556.928561:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533556.928566:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533556.928568:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533556.928569:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533556.928570:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533556.928571:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533556.928573:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533556.928573:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533556.928575:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15356 00000100:00080000:14.0:1584533556.928577:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533556.928586:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533556.928587:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533556.928588:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533556.928589:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533556.928590:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533556.928591:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533556.928591:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533556.928593:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533556.934654:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533556.934658:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533556.934659:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533556.934660:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533556.934662:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533556.934663:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533556.934665:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533556.934667:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533637 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533556.934696:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078401984 new x1661492078402032 00000800:00020000:4.0:1584533562.916139:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533562.916145:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533562.916149:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533562.916630:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533562.928723:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533556/real 1584533556] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533643 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533562.928730:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533562.928731:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533562.928733:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533562.928737:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533562.928739:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533562.928747:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533562.928748:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533562.928749:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533562.928750:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533562.928751:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533562.928752:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15362 00000100:00080000:14.0:1584533562.928765:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533562.928770:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533562.928771:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533562.928772:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533562.928784:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533562.928784:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533562.928786:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533562.928786:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533562.928787:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533562.934867:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533562.934875:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533562.934889:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533562.934890:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533562.934891:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533562.934893:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533562.934894:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533562.934896:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533643 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533562.934930:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078402032 new x1661492078402080 00000800:00020000:2.0:1584533568.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533568.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533568.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533568.916504:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533568.928558:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533562/real 1584533562] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533649 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533568.928567:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533568.928568:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533568.928569:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533568.928584:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533568.928586:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533568.928587:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533568.928588:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533568.928589:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533568.928590:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533568.928591:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533568.928593:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15368 00000100:00080000:14.0:1584533568.928595:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533568.928601:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533568.928603:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533568.928603:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533568.928604:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533568.928605:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533568.928606:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533568.928607:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533568.928608:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533568.934605:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533568.934609:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533568.934611:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533568.934612:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533568.934614:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533568.934616:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533568.934617:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533568.934619:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533649 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533568.934648:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078402080 new x1661492078402144 00000800:00020000:2.0:1584533574.916095:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533574.916101:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533574.916105:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533574.916519:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533574.928607:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533568/real 1584533568] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533655 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533574.928614:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533574.928616:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533574.928617:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533574.928633:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533574.928635:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533574.928636:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533574.928637:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533574.928637:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533574.928639:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533574.928639:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533574.928641:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15374 00000100:00080000:14.0:1584533574.928642:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533574.928650:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533574.928651:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533574.928652:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533574.928653:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533574.928654:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533574.928655:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533574.928655:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533574.928656:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533574.934740:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533574.934745:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533574.934746:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533574.934747:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533574.934748:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533574.934750:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533574.934751:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533574.934753:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533655 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533574.934782:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078402144 new x1661492078402176 00000800:00020000:2.0:1584533580.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533580.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533580.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533580.916498:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533580.928551:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533574/real 1584533574] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533661 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533580.928559:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533580.928560:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533580.928562:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533580.928566:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533580.928568:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533580.928569:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533580.928570:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533580.928571:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533580.928572:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533580.928573:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533580.928574:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15380 00000100:00080000:14.0:1584533580.928576:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533580.928584:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533580.928585:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533580.928586:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533580.928587:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533580.928588:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533580.928589:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533580.928590:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533580.928591:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533580.934449:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533580.934453:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533580.934455:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533580.934455:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533580.934457:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533580.934458:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533580.934460:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533580.934462:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533661 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533580.934493:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078402176 new x1661492078402208 00000800:00020000:6.0:1584533586.916117:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584533586.916120:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584533586.916123:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533586.916548:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533586.928639:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533580/real 1584533580] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533667 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533586.928649:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533586.928651:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533586.928653:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533586.928660:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533586.928662:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533586.928677:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533586.928678:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533586.928679:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533586.928680:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533586.928681:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533586.928694:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15386 00000100:00080000:14.0:1584533586.928697:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533586.928717:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533586.928718:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533586.928719:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533586.928720:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533586.928721:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533586.928722:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533586.928724:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533586.928725:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533589.110934:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533589.110941:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533589.110943:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533589.110944:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533589.110946:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533589.110949:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533589.110951:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533589.110953:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533667 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533589.110983:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078402208 new x1661492078402272 00000800:00000100:0.0:1584533594.140576:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533594.141113:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533594.153191:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533589/real 1584533589] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533678 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533594.153201:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533594.153203:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533594.153204:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533594.153225:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533594.153227:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533594.153228:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533594.153229:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533594.153230:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533594.153232:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533594.153232:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533594.153234:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15392 00000100:00080000:14.0:1584533594.153237:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533594.153244:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533594.153246:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533594.153246:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533594.153248:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533594.153249:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533594.153251:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533594.153251:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533594.153253:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533594.159047:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533594.159054:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533594.159056:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533594.159057:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533594.159059:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533594.159062:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533594.159063:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533594.159065:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533678 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533594.159094:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078402272 new x1661492078402320 00000800:00020000:2.0:1584533600.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533600.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533600.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533600.916509:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533600.928563:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533594/real 1584533594] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533681 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533600.928570:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533600.928572:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533600.928573:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533600.928578:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533600.928579:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533600.928580:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533600.928581:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533600.928582:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533600.928583:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533600.928584:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533600.928585:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15400 00000100:00080000:14.0:1584533600.928588:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533600.928596:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533600.928597:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533600.928598:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533600.928600:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533600.928600:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533600.928601:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533600.928602:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533600.928603:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533600.982502:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533600.982507:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533600.982509:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533600.982510:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533600.982511:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533600.982514:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533600.982515:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533600.982517:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533681 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533600.982547:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078402320 new x1661492078402368 00000800:00000100:0.0:1584533606.141146:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533606.141572:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533606.153622:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533600/real 1584533600] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533687 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533606.153630:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533606.153631:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533606.153633:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533606.153638:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533606.153639:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533606.153640:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533606.153641:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533606.153642:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533606.153643:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533606.153644:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533606.153645:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15406 00000100:00080000:14.0:1584533606.153647:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533606.153656:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533606.153658:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533606.153658:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533606.153659:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533606.153660:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533606.153661:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533606.153662:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533606.153663:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533606.195109:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584533606.195113:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584533606.195115:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584533606.195116:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584533606.195117:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584533606.195119:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584533606.195120:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584533606.195122:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533687 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533606.195151:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078402368 new x1661492078402400 00000800:00000100:0.0:1584533612.141228:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533612.141665:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533612.153735:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533606/real 1584533606] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533693 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533612.153742:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533612.179084:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533612.179085:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533612.179089:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533612.179091:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533612.179098:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533612.179099:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533612.179100:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533612.179101:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533612.179101:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533612.179103:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15412 00000100:00080000:14.0:1584533612.179105:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533612.179121:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533612.179123:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533612.179123:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533612.179124:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533612.179125:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533612.179126:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533612.179127:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533612.179128:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533612.245841:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584533612.245850:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584533612.245853:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584533612.245855:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584533612.245858:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584533612.245862:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584533612.245866:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584533612.245870:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533693 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533612.245917:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078402400 new x1661492078402464 00000800:00000100:0.0:1584533618.141433:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533618.141925:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533618.153974:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533612/real 1584533612] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533699 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533618.153981:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533618.153982:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533618.153984:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533618.153996:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533618.153998:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533618.153998:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533618.154000:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533618.154000:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533618.154002:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533618.154003:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533618.154005:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15418 00000100:00080000:14.0:1584533618.154006:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533618.154012:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533618.154013:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533618.154013:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533618.154015:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533618.154015:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533618.154016:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533618.154017:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533618.154018:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533618.203954:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584533618.203958:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584533618.203960:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584533618.203961:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584533618.203962:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584533618.203964:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584533618.203965:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584533618.221749:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533699 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533618.221782:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078402464 new x1661492078402512 00000800:00020000:2.0:1584533624.916052:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533624.916054:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533624.916056:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533624.916461:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533624.928512:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533618/real 1584533618] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533705 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533624.928520:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533624.928521:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533624.928523:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533624.928527:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533624.928529:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533624.928530:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533624.928532:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533624.928532:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533624.928534:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533624.928534:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533624.928536:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15424 00000100:00080000:14.0:1584533624.928538:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533624.928547:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533624.928548:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533624.928548:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533624.928550:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533624.928550:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533624.928551:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533624.928552:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533624.928553:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533624.981313:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584533624.981317:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584533624.981319:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584533624.981320:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584533624.981321:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584533624.981323:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584533624.981324:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584533624.981326:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533705 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533624.981355:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078402512 new x1661492078402544 00000800:00000100:0.0:1584533630.141798:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533630.142230:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533630.154276:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533624/real 1584533624] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533711 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533630.154283:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533630.154284:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533630.154286:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533630.154290:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533630.154293:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533630.154293:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533630.154295:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533630.154295:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533630.154297:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533630.154297:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533630.154299:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15430 00000100:00080000:14.0:1584533630.154301:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533630.154310:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533630.154311:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533630.154311:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533630.154313:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533630.154313:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533630.154314:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533630.154315:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533630.154316:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533630.186366:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584533630.186371:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584533630.186372:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584533630.186373:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584533630.186374:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584533630.186376:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584533630.186378:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584533630.186380:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533711 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533630.186408:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078402544 new x1661492078402592 00000800:00000100:0.0:1584533636.142056:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533636.142554:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533636.154611:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533630/real 1584533630] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533717 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533636.154619:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533636.154620:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533636.154622:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533636.154626:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533636.154628:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533636.154636:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533636.154637:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533636.154638:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533636.154639:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533636.154640:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533636.154641:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15436 00000100:00080000:14.0:1584533636.154643:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533636.154648:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533636.154650:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533636.154650:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533636.154651:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533636.154652:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533636.154653:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533636.154654:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533636.154655:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533638.888672:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584533638.888676:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584533638.888677:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584533638.888678:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584533638.888679:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584533638.888681:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584533638.888683:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584533638.888684:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533717 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533638.888695:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078402592 new x1661492078402640 00000800:00020000:2.0:1584533643.916058:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533643.916061:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533643.916063:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533643.916452:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533643.928505:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533638/real 1584533638] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533726 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533643.928517:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533643.928519:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533643.928520:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533643.928533:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533643.928535:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533643.928535:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533643.928537:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533643.928537:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533643.928539:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533643.928539:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533643.928541:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15442 00000100:00080000:14.0:1584533643.928543:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533643.928549:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533643.928550:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533643.928551:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533643.928552:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533643.928553:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533643.928554:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533643.928555:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533643.928556:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533643.972630:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584533643.972635:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584533643.972636:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584533643.972638:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584533643.972639:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584533643.972641:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584533643.972643:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584533643.972644:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533726 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533643.972673:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078402640 new x1661492078402704 00000800:00020000:2.0:1584533649.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533649.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533649.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533649.916461:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533649.928518:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533643/real 1584533643] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533730 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533649.928528:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533649.928529:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533649.928531:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533649.928537:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533649.928539:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533649.928540:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533649.928541:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533649.928542:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533649.928543:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533649.928544:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533649.928546:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15449 00000100:00080000:14.0:1584533649.928548:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533649.928562:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533649.928564:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533649.928564:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533649.928566:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533649.928566:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533649.928568:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533649.928569:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533649.928571:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533649.963046:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584533649.963052:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584533649.963054:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584533649.963056:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584533649.963057:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584533649.963060:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584533649.963061:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584533649.963063:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533730 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533649.963093:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078402704 new x1661492078402736 00000800:00000100:0.0:1584533656.141709:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533656.142149:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533656.154196:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533649/real 1584533649] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533736 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533656.154206:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533656.154208:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533656.154210:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533656.154217:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533656.154219:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533656.154220:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533656.154221:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533656.154222:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533656.154223:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533656.154224:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533656.154226:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15455 00000100:00080000:14.0:1584533656.154229:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533656.154244:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533656.154246:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533656.154246:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533656.154248:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533656.154249:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533656.154251:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533656.154251:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533656.154253:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533656.197950:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584533656.197957:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584533656.197959:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584533656.197960:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584533656.197962:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584533656.197965:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584533656.197966:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584533656.197968:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533736 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533656.197999:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078402736 new x1661492078402768 00000800:00020000:4.0:1584533661.916137:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533661.916140:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533661.916142:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533661.916583:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533661.928636:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533656/real 1584533656] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533743 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533661.928645:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533661.928646:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533661.928647:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533661.928652:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533661.928654:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533661.928661:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533661.928663:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533661.928663:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533661.928665:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533661.928666:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533661.928667:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15462 00000100:00080000:14.0:1584533661.928670:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533661.928676:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533661.928677:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533661.928679:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533661.928680:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533661.928681:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533661.928682:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533661.928682:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533661.928684:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533661.981920:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584533661.981926:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584533661.981927:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584533661.981929:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584533661.981930:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584533661.981932:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584533661.981934:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584533661.981936:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533743 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533661.981966:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078402768 new x1661492078402832 00000800:00020000:4.0:1584533667.916180:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533667.916183:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533667.916185:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533667.916620:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533667.928676:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533661/real 1584533661] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533748 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533667.928685:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533667.928686:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533667.928688:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533667.928703:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533667.928705:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533667.928705:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533667.928707:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533667.928707:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533667.928709:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533667.928710:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533667.928712:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15467 00000100:00080000:14.0:1584533667.928714:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533667.928722:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533667.928723:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533667.928724:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533667.928725:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533667.928726:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533667.928727:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533667.928728:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533667.928729:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533667.988567:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584533667.988573:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584533667.988575:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584533667.988576:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584533667.988578:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584533667.988580:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584533667.988582:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584533667.988584:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533748 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533667.988615:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078402832 new x1661492078402880 00000800:00020000:2.0:1584533673.916118:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533673.916120:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533673.916122:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533673.916509:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533673.928561:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533667/real 1584533667] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533754 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533673.928569:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533673.928570:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533673.928572:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533673.928577:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533673.928579:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533673.928580:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533673.928581:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533673.928582:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533673.928583:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533673.928584:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533673.928587:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15473 00000100:00080000:14.0:1584533673.928589:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533673.928598:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533673.928599:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533673.928600:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533673.928601:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533673.928602:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533673.928603:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533673.928604:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533673.928605:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533674.013714:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584533674.013719:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584533674.013721:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584533674.013722:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584533674.013724:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584533674.013726:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584533674.013728:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584533674.013729:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533754 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533674.013763:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078402880 new x1661492078402928 00000800:00000100:0.0:1584533680.141314:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533680.141721:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533680.153769:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533674/real 1584533674] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533761 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533680.153777:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533680.153778:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533680.153779:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533680.153784:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533680.153786:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533680.153786:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533680.153788:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533680.153788:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533680.153790:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533680.153791:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533680.153793:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15479 00000100:00080000:14.0:1584533680.153795:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533680.153803:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533680.153804:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533680.153805:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533680.153806:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533680.153807:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533680.153808:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533680.153808:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533680.153810:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533680.186803:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584533680.186807:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584533680.186809:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584533680.186810:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584533680.186811:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584533680.186813:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584533680.186814:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584533680.186816:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533761 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533680.186847:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078402928 new x1661492078402960 00000800:00020000:4.0:1584533685.916132:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533685.916135:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533685.916136:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533685.916664:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533685.928717:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533680/real 1584533680] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533767 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533685.928724:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533685.928725:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533685.928727:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533685.928733:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533685.928734:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533685.928743:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533685.928745:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533685.928746:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533685.928747:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533685.928748:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533685.928750:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15486 00000100:00080000:14.0:1584533685.928752:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533685.928757:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533685.928758:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533685.928759:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533685.928760:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533685.928761:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533685.928762:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533685.928762:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533685.928763:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533685.980023:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584533685.980039:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584533685.980041:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584533685.980041:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584533685.980043:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584533685.980045:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584533685.980046:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584533685.980048:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533767 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533685.980079:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078402960 new x1661492078403008 00000800:00020000:4.0:1584533691.916132:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533691.916135:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533691.916136:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533691.916608:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533691.928660:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533685/real 1584533685] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533772 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533691.928668:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533691.928669:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533691.928671:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533691.928684:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533691.928687:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533691.928688:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533691.928689:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533691.928690:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533691.928691:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533691.928692:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533691.928693:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15491 00000100:00080000:14.0:1584533691.928696:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533691.928701:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533691.928703:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533691.928703:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533691.928705:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533691.928705:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533691.928706:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533691.928707:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533691.928708:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533691.959803:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584533691.959808:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584533691.959809:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584533691.959810:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584533691.959812:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584533691.959814:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584533691.959815:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584533691.959817:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533772 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533691.959847:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078403008 new x1661492078403072 00000800:00020000:4.0:1584533697.916202:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533697.916205:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533697.916207:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533697.916703:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533697.928754:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533691/real 1584533691] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533778 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533697.928762:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533697.928763:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533697.928765:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533697.928771:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533697.928772:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533697.928773:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533697.928775:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533697.928775:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533697.928777:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533697.928777:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533697.928779:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15497 00000100:00080000:14.0:1584533697.928781:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533697.928789:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533697.928790:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533697.928790:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533697.928792:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533697.928792:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533697.928793:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533697.928794:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533697.928795:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533697.973965:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584533697.973969:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584533697.973971:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584533697.973972:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584533697.973973:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584533697.973975:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584533697.973976:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584533697.973978:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533778 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533697.974008:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078403072 new x1661492078403104 00000800:00000100:0.0:1584533704.142083:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533704.142531:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533704.154580:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533697/real 1584533697] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533784 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533704.154593:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533704.154594:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533704.154596:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533704.154602:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533704.154603:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533704.154604:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533704.154605:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533704.154606:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533704.154607:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533704.154608:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533704.154610:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15503 00000100:00080000:14.0:1584533704.154613:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533704.154622:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533704.154623:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533704.154624:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533704.154625:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533704.154625:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533704.154627:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533704.154627:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533704.154628:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533704.207693:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584533704.207697:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584533704.207699:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584533704.207700:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584533704.207702:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584533704.207704:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584533704.207705:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584533704.207707:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533784 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533704.207737:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078403104 new x1661492078403152 00000800:00020000:2.0:1584533709.916062:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533709.916065:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533709.916068:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533709.916649:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533709.928711:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533704/real 1584533704] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533791 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533709.928721:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533709.928723:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533709.928724:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533709.928732:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533709.928734:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533709.928747:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533709.928749:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533709.928750:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533709.928751:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533709.928752:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533709.928754:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15510 00000100:00080000:14.0:1584533709.928757:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533709.928765:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533709.928766:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533709.928767:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533709.928768:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533709.928769:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533709.928770:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533709.928772:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533709.928773:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533709.985280:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584533709.985286:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584533709.985289:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584533709.985290:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584533709.985291:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584533709.985294:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584533709.985295:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584533709.985297:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533791 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533709.985326:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078403152 new x1661492078403200 00000800:00020000:2.0:1584533715.916040:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533715.916043:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533715.916045:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533715.916460:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533715.928513:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533709/real 1584533709] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533796 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533715.928522:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533715.928524:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533715.928525:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533715.928546:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533715.928548:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533715.928549:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533715.928551:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533715.928551:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533715.928553:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533715.928554:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533715.928556:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15515 00000100:00080000:14.0:1584533715.928558:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533715.928566:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533715.928567:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533715.928568:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533715.928569:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533715.928571:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533715.928572:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533715.928573:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533715.928574:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533715.981238:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584533715.981245:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584533715.981247:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584533715.981248:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584533715.981250:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584533715.981252:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584533715.981254:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584533715.981256:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533796 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533715.981285:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078403200 new x1661492078403248 00000800:00020000:6.0:1584533721.916070:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584533721.916073:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584533721.916075:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533721.916519:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533721.928577:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533715/real 1584533715] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533802 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533721.928586:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533721.928588:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533721.928589:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533721.928596:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533721.928598:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533721.928599:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533721.928600:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533721.928601:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533721.928602:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533721.928603:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533721.928605:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15521 00000100:00080000:14.0:1584533721.928608:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533721.928619:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533721.928621:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533721.928623:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533721.928624:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533721.928625:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533721.928626:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533721.928627:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533721.928628:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533721.980726:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584533721.980733:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584533721.980735:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584533721.980736:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584533721.980737:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584533721.980740:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584533721.980741:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584533721.980744:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533802 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533721.980775:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078403248 new x1661492078403296 00000800:00000100:0.0:1584533728.140862:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533728.141334:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533728.153387:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533721/real 1584533721] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533808 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533728.153395:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533728.153396:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533728.153398:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533728.153404:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533728.153406:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533728.153407:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533728.153409:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533728.153409:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533728.153411:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533728.153412:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533728.153413:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15527 00000100:00080000:14.0:1584533728.153415:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533728.153428:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533728.153430:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533728.153431:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533728.153432:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533728.153432:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533728.153434:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533728.153435:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533728.153436:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533728.204306:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584533728.204311:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584533728.204313:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584533728.204314:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584533728.204316:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584533728.204318:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584533728.204319:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584533728.204321:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533808 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533728.204351:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078403296 new x1661492078403328 00000800:00020000:6.0:1584533733.916192:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584533733.916195:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584533733.916197:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533733.916704:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533733.928760:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533728/real 1584533728] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533815 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533733.928768:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533733.928770:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533733.928771:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533733.928777:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533733.928779:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533733.928789:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533733.928791:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533733.928791:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533733.928793:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533733.928794:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533733.928796:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15534 00000100:00080000:14.0:1584533733.928799:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533733.928805:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533733.928806:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533733.928807:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533733.928808:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533733.928809:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533733.928810:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533733.928811:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533733.928812:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533733.982974:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584533733.982978:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584533733.982980:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584533733.982980:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584533733.982982:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584533733.982984:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584533733.982986:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584533733.982987:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533815 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533733.983016:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078403328 new x1661492078403392 00000800:00020000:2.0:1584533739.916056:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533739.916058:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533739.916060:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533739.916473:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533739.928525:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533733/real 1584533733] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533820 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533739.928532:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533739.928533:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533739.928534:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533739.928548:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533739.928549:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533739.928550:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533739.928551:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533739.928552:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533739.928554:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533739.928555:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533739.928556:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15539 00000100:00080000:14.0:1584533739.928558:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533739.928563:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533739.928564:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533739.928565:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533739.928566:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533739.928567:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533739.928568:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533739.928568:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533739.928570:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533739.987302:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584533739.987306:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584533739.987308:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584533739.987309:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584533739.987311:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584533739.987312:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584533739.987314:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584533739.987316:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533820 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533739.987325:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078403392 new x1661492078403440 00000800:00020000:2.0:1584533745.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533745.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533745.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533745.916501:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533745.928558:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533739/real 1584533739] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533826 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533745.928568:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533745.928569:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533745.928571:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533745.928579:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533745.928581:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533745.928582:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533745.928584:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533745.928585:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533745.928586:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533745.928587:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533745.928589:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15545 00000100:00080000:14.0:1584533745.928592:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533745.928607:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533745.928609:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533745.928609:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533745.928611:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533745.928611:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533745.928612:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533745.928613:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533745.928614:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533745.990336:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584533745.990342:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584533745.990344:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584533745.990345:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584533745.990347:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584533745.990350:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584533745.990351:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584533745.990353:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533826 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533745.990385:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078403440 new x1661492078403472 00000800:00000100:0.0:1584533752.141604:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533752.141997:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533752.154049:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533745/real 1584533745] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533832 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533752.154058:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533752.154060:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533752.154061:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533752.154069:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533752.154072:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533752.154072:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533752.154074:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533752.154075:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533752.154076:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533752.154077:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533752.154079:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15551 00000100:00080000:14.0:1584533752.154081:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533752.154092:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533752.154093:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533752.154094:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533752.154095:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533752.154096:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533752.154097:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533752.154097:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533752.154099:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533752.208250:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584533752.208256:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584533752.208258:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584533752.208259:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584533752.208260:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584533752.208263:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584533752.208264:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584533752.208266:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533832 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533752.208295:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078403472 new x1661492078403520 00000800:00020000:2.0:1584533757.916172:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533757.938985:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533757.963160:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533757.963594:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533757.975671:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533752/real 1584533752] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533839 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533757.975678:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533757.975680:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533757.975682:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533757.975688:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533757.975689:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533757.975698:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533757.975700:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533757.975700:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533757.975702:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533757.975702:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533757.975704:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15558 00000100:00080000:14.0:1584533757.975706:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533757.975711:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533757.975712:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533757.975713:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533757.975714:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533757.975715:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533757.975716:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533757.975717:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533757.975718:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533758.014702:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584533758.014707:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584533758.014708:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584533758.014709:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584533758.014710:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584533758.014712:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584533758.014714:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584533758.014715:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533839 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533758.014745:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078403520 new x1661492078403568 00000800:00020000:2.0:1584533763.916128:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533763.916132:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533763.916134:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533763.916543:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533763.928641:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533758/real 1584533758] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533845 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533763.928655:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533763.928657:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533763.928658:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533763.928677:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533763.928679:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533763.928680:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533763.928681:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533763.928682:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533763.928684:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533763.928685:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533763.928687:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15563 00000100:00080000:14.0:1584533763.928690:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533763.928696:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533763.928698:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533763.928698:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533763.928699:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533763.928700:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533763.928701:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533763.928702:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533763.928703:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533763.981527:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584533763.981533:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584533763.981535:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584533763.981536:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584533763.981538:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584533763.981540:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584533763.981542:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584533763.981544:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533845 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533763.981572:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078403568 new x1661492078403632 00000800:00020000:2.0:1584533769.916062:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533769.916065:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533769.916067:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533769.916503:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533769.928555:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533763/real 1584533763] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533850 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533769.928564:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533769.928566:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533769.928567:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533769.928574:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533769.928576:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533769.928577:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533769.928578:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533769.928579:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533769.928581:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533769.928582:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533769.928583:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15569 00000100:00080000:14.0:1584533769.928586:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533769.928600:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533769.928601:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533769.928602:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533769.928603:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533769.928604:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533769.928605:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533769.928607:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533769.928608:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533769.992404:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584533769.992409:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584533769.992412:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584533769.992413:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584533769.992415:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584533769.992418:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584533769.992419:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584533769.992421:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533850 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533769.992451:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078403632 new x1661492078403664 00000800:00000100:0.0:1584533776.141163:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533776.141637:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533776.153696:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533769/real 1584533769] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533856 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533776.153704:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533776.153706:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533776.153707:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533776.153714:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533776.153716:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533776.153717:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533776.153718:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533776.153719:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533776.153720:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533776.153721:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533776.153723:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15575 00000100:00080000:14.0:1584533776.153725:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533776.153739:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533776.153740:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533776.153741:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533776.153742:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533776.153744:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533776.153745:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533776.153746:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533776.153747:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533776.219696:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584533776.219702:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584533776.219704:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584533776.219705:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584533776.219707:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584533776.219709:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584533776.219710:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584533776.219712:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533856 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533776.219741:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078403664 new x1661492078403696 00000800:00020000:10.0:1584533781.916203:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:10.0:1584533781.916206:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:10.0:1584533781.916208:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533781.916599:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533781.928656:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533776/real 1584533776] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533863 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533781.928664:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533781.928665:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533781.928666:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533781.928672:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533781.928674:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533781.928683:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533781.928685:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533781.928685:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533781.928687:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533781.928688:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533781.928690:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15582 00000100:00080000:14.0:1584533781.928692:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533781.928698:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533781.928699:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533781.928701:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533781.928702:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533781.928703:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533781.928704:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533781.928705:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533781.928706:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533781.974265:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584533781.974270:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584533781.974271:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584533781.974272:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584533781.974273:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584533781.974275:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584533781.974277:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584533781.974278:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533863 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584533781.974310:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078403696 new x1661492078403760 00000800:00020000:8.0:1584533787.916216:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584533787.916219:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584533787.916221:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533787.916657:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584533787.928712:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533781/real 1584533781] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533868 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584533787.928720:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584533787.928721:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584533787.928722:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584533787.928736:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533787.928737:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533787.928738:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533787.928739:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533787.928740:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533787.928741:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584533787.928742:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584533787.928743:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15587 00000100:00080000:14.0:1584533787.928746:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584533787.928752:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533787.928754:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533787.928754:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533787.928756:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533787.928756:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533787.928757:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533787.928758:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584533787.928759:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533787.981315:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584533787.981319:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584533787.981321:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584533787.981322:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584533787.981323:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584533787.981325:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584533787.981327:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584533787.981328:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533868 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0F:1584533787.981358:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078403760 new x1661492078403808 00000800:00020000:2.0:1584533793.916117:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533793.916119:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533793.916121:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533793.916591:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584533793.928646:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533787/real 1584533787] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533874 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584533793.928654:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584533793.928655:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584533793.928656:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584533793.928662:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584533793.928663:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584533793.928664:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584533793.928665:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584533793.928666:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584533793.928667:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584533793.928668:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584533793.928669:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15593 00000100:00080000:16.0:1584533793.928671:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584533793.928681:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584533793.928682:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584533793.928683:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584533793.928684:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584533793.928684:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584533793.928686:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584533793.928686:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584533793.928688:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533793.988117:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584533793.988121:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584533793.988122:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584533793.988123:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584533793.988124:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584533793.988126:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584533793.988128:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584533793.988129:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533874 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584533793.988158:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078403808 new x1661492078403856 00000800:00000100:0.0:1584533800.142014:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533800.142447:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584533800.154497:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533793/real 1584533793] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533880 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584533800.154505:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584533800.154507:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584533800.154508:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584533800.154513:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584533800.154515:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584533800.154516:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584533800.154517:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584533800.154518:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584533800.154519:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584533800.154520:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584533800.154521:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15599 00000100:00080000:16.0:1584533800.154523:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584533800.154532:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584533800.154533:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584533800.154533:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584533800.154535:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584533800.154535:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584533800.154538:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584533800.154538:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584533800.154539:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533800.197256:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584533800.197260:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584533800.197261:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584533800.197262:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584533800.197264:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584533800.197265:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584533800.197267:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584533800.197269:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533880 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584533800.197298:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078403856 new x1661492078403888 00000800:00020000:2.0:1584533805.916069:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533805.916072:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533805.916073:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533805.916545:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584533805.928598:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533800/real 1584533800] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533887 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584533805.928606:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584533805.928607:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584533805.928609:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584533805.928615:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584533805.928617:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584533805.928624:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584533805.928626:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584533805.928626:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584533805.928628:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584533805.928629:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584533805.928630:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15606 00000100:00080000:16.0:1584533805.928632:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584533805.928637:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584533805.928639:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584533805.928639:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584533805.928641:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584533805.928642:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584533805.928643:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584533805.928644:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584533805.928645:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533805.980128:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584533805.980132:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584533805.980133:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584533805.980134:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584533805.980136:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584533805.980138:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584533805.980139:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584533805.980141:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533887 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584533805.980169:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078403888 new x1661492078403936 00000800:00020000:4.0:1584533811.916088:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533811.916091:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533811.916093:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533811.916535:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584533811.928589:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533805/real 1584533805] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533892 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584533811.928597:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584533811.928599:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584533811.928600:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584533811.928614:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533811.928616:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533811.928617:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533811.928618:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533811.928618:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533811.928620:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584533811.928620:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584533811.928622:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15611 00000100:00080000:18.0:1584533811.928625:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584533811.928631:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533811.928633:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533811.928633:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533811.928634:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533811.928635:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533811.928636:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533811.928637:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533811.928638:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533811.980656:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584533811.980661:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584533811.980662:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584533811.980663:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584533811.980664:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584533811.980666:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584533811.980668:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584533811.980670:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533892 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584533811.980699:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078403936 new x1661492078404000 00000800:00020000:2.0:1584533817.916072:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533817.916075:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533817.916076:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533817.916488:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584533817.928539:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533811/real 1584533811] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533898 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584533817.928547:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584533817.928548:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584533817.928549:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584533817.928554:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533817.928556:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533817.928557:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533817.928558:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533817.928558:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533817.928560:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584533817.928561:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584533817.928562:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15617 00000100:00080000:18.0:1584533817.928564:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584533817.928572:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533817.928574:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533817.928574:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533817.928575:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533817.928576:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533817.928577:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533817.928578:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533817.928579:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533817.983180:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584533817.983184:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584533817.983186:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584533817.983187:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584533817.983189:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584533817.983190:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584533817.983192:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584533817.983194:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533898 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584533817.983223:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078404000 new x1661492078404032 00000800:00000100:0.0:1584533824.142913:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533824.143533:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584533824.155608:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533817/real 1584533817] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533904 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584533824.155618:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584533824.155619:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584533824.155621:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584533824.155629:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533824.155631:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533824.155632:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533824.155633:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533824.155635:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533824.155636:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584533824.155637:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584533824.155640:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15623 00000100:00080000:18.0:1584533824.155643:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584533824.155656:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533824.155658:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533824.155659:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533824.155660:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533824.155660:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533824.155662:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533824.155662:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533824.155664:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584533824.213906:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584533824.213912:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584533824.213914:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584533824.213915:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584533824.213918:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584533824.213920:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584533824.213922:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584533824.213924:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533904 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584533824.213953:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078404032 new x1661492078404080 00000800:00020000:4.0:1584533829.916056:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533829.916060:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533829.916062:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533829.916588:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584533829.928636:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533824/real 1584533824] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533911 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584533829.928645:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584533829.928647:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584533829.928648:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584533829.928655:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533829.928657:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533829.928670:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533829.928671:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533829.928672:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533829.928673:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584533829.928674:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584533829.928676:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15630 00000100:00080000:18.0:1584533829.928679:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584533829.928685:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533829.928687:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533829.928688:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533829.928689:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533829.928690:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533829.928691:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533829.928692:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533829.928693:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533829.980741:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584533829.980747:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584533829.980749:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584533829.980751:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584533829.980752:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584533829.980755:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584533829.980757:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584533829.980759:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533911 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584533829.980790:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078404080 new x1661492078404128 00000800:00020000:2.0:1584533835.916064:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533835.916067:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533835.916069:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533835.916544:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584533835.928623:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533829/real 1584533829] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533916 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584533835.928631:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584533835.928633:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584533835.928634:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584533835.928652:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533835.928654:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533835.928655:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533835.928656:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533835.928657:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533835.928658:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584533835.928659:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584533835.928661:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15635 00000100:00080000:18.0:1584533835.928663:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584533835.928670:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533835.928671:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533835.928672:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533835.928673:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533835.928674:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533835.928675:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533835.928676:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533835.928677:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533837.228914:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584533837.228920:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584533837.228922:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584533837.228923:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584533837.228925:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584533837.228927:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584533837.228929:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584533837.228931:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533916 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584533837.228963:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078404128 new x1661492078404192 00000800:00020000:14.0:1584533842.916094:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:14.0:1584533842.916097:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:14.0:1584533842.916099:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533842.916562:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584533842.928653:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533837/real 1584533837] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533925 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584533842.928671:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584533842.928673:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584533842.928675:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584533842.928681:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533842.928682:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533842.928683:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533842.928684:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533842.928685:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533842.928686:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584533842.928687:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584533842.928689:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15641 00000100:00080000:18.0:1584533842.928691:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584533842.928700:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533842.928702:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533842.928702:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533842.928703:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533842.928704:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533842.928705:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533842.928706:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533842.928707:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533842.986666:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584533842.986670:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584533842.986672:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584533842.986673:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584533842.986675:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584533842.986676:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584533842.986678:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584533842.986679:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533925 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584533842.986715:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078404192 new x1661492078404224 00000800:00000100:0.0:1584533849.141206:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533849.141638:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584533849.153686:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533842/real 1584533842] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533929 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584533849.153694:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584533849.153695:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584533849.153697:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584533849.153702:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533849.153703:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533849.153704:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533849.153705:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533849.153706:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533849.153707:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584533849.153708:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584533849.153709:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15648 00000100:00080000:18.0:1584533849.153711:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584533849.153719:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533849.153720:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533849.153721:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533849.153723:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533849.153723:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533849.153724:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533849.153725:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533849.153726:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584533849.205768:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584533849.205772:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584533849.205774:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584533849.205775:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584533849.205776:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584533849.205778:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584533849.205779:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584533849.205781:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533929 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584533849.205810:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078404224 new x1661492078404256 00000800:00020000:2.0:1584533854.916067:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533854.916070:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533854.916071:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533854.916479:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584533854.928532:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533849/real 1584533849] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533936 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584533854.928539:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584533854.928540:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584533854.928542:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584533854.928547:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533854.928548:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533854.928557:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533854.928558:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533854.928559:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533854.928560:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584533854.928561:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584533854.928563:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15655 00000100:00080000:18.0:1584533854.928565:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584533854.928570:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533854.928571:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533854.928572:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533854.928573:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533854.928574:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533854.928575:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533854.928576:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533854.928578:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533854.971982:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584533854.971991:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584533854.971994:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584533854.971995:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584533854.971998:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584533854.972016:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584533854.972017:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584533854.972019:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533936 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584533854.972049:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078404256 new x1661492078404320 00000800:00020000:2.0:1584533860.916170:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533860.916173:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533860.916175:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533860.916672:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584533860.928720:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533854/real 1584533854] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533941 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584533860.928728:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584533860.928730:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584533860.928731:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584533860.928743:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533860.928745:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533860.928746:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533860.928747:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533860.928748:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533860.928749:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584533860.928750:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584533860.928751:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15660 00000100:00080000:18.0:1584533860.928753:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584533860.928759:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533860.928760:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533860.928761:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533860.928762:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533860.928763:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533860.928764:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533860.928765:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533860.928766:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533860.982958:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584533860.982962:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584533860.982963:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584533860.982964:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584533860.982965:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584533860.982967:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584533860.982969:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584533860.982970:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533941 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584533860.982999:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078404320 new x1661492078404368 00000800:00020000:4.0:1584533866.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533866.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533866.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533866.916518:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584533866.928583:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533860/real 1584533860] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533947 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584533866.928591:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584533866.928592:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584533866.928593:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584533866.928599:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533866.928600:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533866.928601:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533866.928602:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533866.928603:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533866.928604:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584533866.928605:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584533866.928606:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15666 00000100:00080000:18.0:1584533866.928608:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584533866.928617:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533866.928619:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533866.928620:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533866.928621:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533866.928622:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533866.928623:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533866.928623:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533866.928625:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533866.995765:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584533866.995770:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584533866.995771:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584533866.995772:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584533866.995774:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584533866.995776:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584533866.995777:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584533866.995779:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533947 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584533866.995789:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078404368 new x1661492078404416 00000800:00000100:0.0:1584533873.142671:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533873.143134:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584533873.155183:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533866/real 1584533866] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533953 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584533873.155191:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584533873.155192:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584533873.155193:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584533873.155198:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533873.155200:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533873.155201:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533873.155202:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533873.155203:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533873.155204:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584533873.155204:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584533873.155206:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15672 00000100:00080000:18.0:1584533873.155208:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584533873.155218:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533873.155219:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533873.155220:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533873.155221:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533873.155222:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533873.155223:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533873.155223:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533873.155224:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533873.200956:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584533873.200960:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584533873.200961:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584533873.200962:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584533873.200963:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584533873.200966:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584533873.200967:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584533873.200969:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533953 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584533873.201000:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078404416 new x1661492078404448 00000800:00020000:2.0:1584533878.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533878.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533878.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533878.916520:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584533878.928570:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533873/real 1584533873] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533960 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584533878.928578:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584533878.928579:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584533878.928581:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584533878.928586:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533878.928587:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533878.928596:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533878.928597:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533878.928598:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533878.928599:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584533878.928600:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584533878.928602:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15679 00000100:00080000:18.0:1584533878.928604:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584533878.928609:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533878.928611:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533878.928611:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533878.928612:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533878.928613:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533878.928614:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533878.928615:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533878.928616:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533878.983335:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584533878.983338:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584533878.983340:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584533878.983341:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584533878.983342:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584533878.983344:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584533878.983345:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584533878.983347:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533960 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584533878.983376:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078404448 new x1661492078404496 00000800:00020000:4.0:1584533884.916054:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533884.916058:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533884.916060:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533884.916514:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584533884.928570:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533878/real 1584533878] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533965 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584533884.928580:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584533884.928582:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584533884.928584:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584533884.928602:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533884.928604:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533884.928605:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533884.928606:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533884.928607:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533884.928609:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584533884.928611:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584533884.928613:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15684 00000100:00080000:18.0:1584533884.928616:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584533884.928623:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533884.928624:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533884.928625:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533884.928626:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533884.928627:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533884.928628:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533884.928629:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533884.928630:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533884.978496:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584533884.978501:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584533884.978503:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584533884.978504:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584533884.978506:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584533884.978509:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584533884.978510:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584533884.978512:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533965 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584533884.978541:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078404496 new x1661492078404560 00000800:00020000:4.0:1584533890.916084:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533890.916087:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533890.916089:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533890.916493:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584533890.928553:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533884/real 1584533884] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533971 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584533890.928563:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584533890.928565:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584533890.928567:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584533890.928575:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533890.928578:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533890.928578:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533890.928581:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533890.928582:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533890.928583:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584533890.928584:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584533890.928586:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15690 00000100:00080000:18.0:1584533890.928589:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584533890.928606:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533890.928607:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533890.928608:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533890.928609:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533890.928610:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533890.928611:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533890.928612:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533890.928613:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533890.964181:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584533890.964187:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584533890.964189:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584533890.964191:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584533890.964193:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584533890.964196:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584533890.964197:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584533890.964199:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533971 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584533890.964231:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078404560 new x1661492078404592 00000800:00000100:0.0:1584533897.140925:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533897.141414:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584533897.153485:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533890/real 1584533890] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533977 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584533897.153505:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584533897.153506:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584533897.153508:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584533897.153515:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533897.153518:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533897.153519:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533897.153520:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533897.153521:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533897.153522:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584533897.153523:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584533897.153525:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15696 00000100:00080000:18.0:1584533897.153528:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584533897.153539:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533897.153541:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533897.153542:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533897.153543:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533897.153543:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533897.153545:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533897.153545:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533897.153547:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584533897.211283:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584533897.211293:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584533897.211297:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584533897.211299:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584533897.211302:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584533897.211307:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584533897.211311:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584533897.211315:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533977 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584533897.211364:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078404592 new x1661492078404640 00000800:00020000:4.0:1584533902.916082:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533902.916085:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533902.916087:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533902.916494:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584533902.928556:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533897/real 1584533897] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533984 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584533902.928564:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584533902.928565:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584533902.928567:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584533902.928571:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533902.928573:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533902.928583:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533902.928584:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533902.928585:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533902.928586:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584533902.928587:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584533902.928589:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15703 00000100:00080000:18.0:1584533902.928591:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584533902.928597:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533902.928598:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533902.928599:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533902.928600:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533902.928601:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533902.928602:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533902.928602:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533902.928604:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533902.970179:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533902.970184:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533902.970186:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533902.970186:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533902.970188:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533902.970190:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533902.970191:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533902.970193:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533984 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584533902.970222:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078404640 new x1661492078404688 00000800:00020000:6.0:1584533908.916155:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584533908.916158:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584533908.916160:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533908.916664:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584533908.928718:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533902/real 1584533902] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533989 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584533908.928730:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584533908.928732:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584533908.928733:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584533908.928747:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533908.928749:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533908.928750:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533908.928751:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533908.928751:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533908.928753:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584533908.928754:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584533908.928756:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15708 00000100:00080000:18.0:1584533908.928758:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584533908.928764:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533908.928765:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533908.928766:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533908.928767:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533908.928768:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533908.928769:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533908.928769:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533908.928771:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533908.969414:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533908.969419:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533908.969420:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533908.969421:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533908.969423:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533908.969425:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533908.969426:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533908.969428:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533989 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584533908.969457:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078404688 new x1661492078404736 00000800:00020000:2.0:1584533914.916063:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533914.916065:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533914.916067:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533914.916505:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584533914.928559:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533908/real 1584533908] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533995 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584533914.928567:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584533914.928568:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584533914.928569:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584533914.928575:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533914.928576:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533914.928577:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533914.928578:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533914.928579:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533914.928581:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584533914.928581:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584533914.928583:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15714 00000100:00080000:18.0:1584533914.928585:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584533914.928594:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533914.928595:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533914.928596:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533914.928597:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533914.928598:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533914.928599:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533914.928600:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533914.928601:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533914.968351:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533914.968355:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533914.968356:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533914.968357:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533914.968358:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533914.968360:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533914.968362:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533914.968364:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584533995 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584533914.968393:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078404736 new x1661492078404784 00000800:00000100:0.0:1584533921.141104:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533921.141539:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584533921.153587:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533914/real 1584533914] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534001 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584533921.153595:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584533921.153596:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584533921.153597:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584533921.153602:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533921.153604:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533921.153604:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533921.153605:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533921.153606:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533921.153607:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584533921.153608:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584533921.153610:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15720 00000100:00080000:18.0:1584533921.153612:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584533921.153620:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533921.153621:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533921.153622:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533921.153623:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533921.153624:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533921.153625:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533921.153626:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533921.153627:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533921.159451:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533921.159454:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533921.159456:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533921.159457:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533921.159458:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533921.159460:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533921.159461:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533921.159463:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534001 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584533921.159491:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078404784 new x1661492078404816 00000800:00020000:2.0:1584533926.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533926.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533926.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533926.916519:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584533926.928586:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533921/real 1584533921] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534008 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584533926.928594:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584533926.928595:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584533926.928596:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584533926.928602:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533926.928604:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533926.928613:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533926.928615:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533926.928615:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533926.928617:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584533926.928617:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584533926.928619:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15727 00000100:00080000:18.0:1584533926.928621:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584533926.928626:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533926.928628:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533926.928629:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533926.928631:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533926.928631:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533926.928632:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533926.928633:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533926.928634:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533926.934534:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533926.934539:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533926.934541:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533926.934541:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533926.934543:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533926.934545:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533926.934547:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533926.934549:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534008 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584533926.934558:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078404816 new x1661492078404880 00000800:00020000:4.0:1584533932.916059:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533932.916061:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533932.916063:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533932.916436:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584533932.928487:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533926/real 1584533926] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534013 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584533932.928495:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584533932.928496:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584533932.928498:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584533932.928511:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533932.928512:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533932.928513:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533932.928514:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533932.928515:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533932.928516:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584533932.928517:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584533932.928518:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15732 00000100:00080000:18.0:1584533932.928520:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584533932.928526:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533932.928528:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533932.928528:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533932.928530:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533932.928530:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533932.928531:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584533932.928532:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584533932.928533:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533932.934640:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533932.934643:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533932.934645:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533932.934646:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533932.934647:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533932.934649:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533932.934650:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533932.934652:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534013 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584533932.934681:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078404880 new x1661492078404928 00000800:00020000:2.0:1584533938.916059:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533938.916062:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533938.916064:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533938.916480:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584533938.928535:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533932/real 1584533932] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534019 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584533938.928542:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584533938.928544:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584533938.928545:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584533938.928551:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533938.928553:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533938.928554:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533938.928556:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533938.928556:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533938.928558:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584533938.928558:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584533938.928560:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15738 00000100:00080000:20.0:1584533938.928562:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584533938.928572:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533938.928573:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533938.928574:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533938.928575:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533938.928575:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533938.928577:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533938.928577:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533938.928578:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533938.934455:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533938.934459:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533938.934460:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533938.934461:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533938.934463:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533938.934464:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533938.934466:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533938.934467:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534019 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584533938.934497:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078404928 new x1661492078404960 00000800:00000100:0.0:1584533945.142014:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533945.142559:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584533945.154615:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533938/real 1584533938] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534025 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584533945.154624:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584533945.154625:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584533945.154627:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584533945.154633:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533945.154635:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533945.154635:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533945.154636:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533945.154637:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533945.154639:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584533945.154639:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584533945.154641:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15744 00000100:00080000:20.0:1584533945.154643:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584533945.154652:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533945.154653:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533945.154654:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533945.154655:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533945.154656:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533945.154657:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533945.154658:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533945.154659:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533945.160553:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533945.160558:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533945.160560:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533945.160561:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533945.160562:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533945.160564:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533945.160565:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533945.160567:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534025 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584533945.160596:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078404960 new x1661492078405008 00000800:00020000:2.0:1584533950.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533950.916081:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533950.916083:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533950.916584:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584533950.928661:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533945/real 1584533945] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534032 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584533950.928674:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584533950.928675:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584533950.928676:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584533950.928682:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533950.928684:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533950.928693:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533950.928694:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533950.928695:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533950.928696:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584533950.928697:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584533950.928698:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15751 00000100:00080000:20.0:1584533950.928700:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584533950.928706:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533950.928708:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533950.928708:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533950.928710:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533950.928710:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533950.928711:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584533950.928712:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584533950.928713:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584533950.934815:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584533950.934824:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584533950.934827:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584533950.934830:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584533950.934832:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584533950.934837:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584533950.934840:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584533950.934844:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534032 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584533950.934897:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078405008 new x1661492078405056 00000800:00020000:2.0:1584533956.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533956.916081:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533956.916083:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533956.916495:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584533956.928586:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533950/real 1584533950] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534037 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584533956.928596:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584533956.928598:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584533956.928599:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584533956.928621:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533956.928623:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533956.928624:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533956.928626:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533956.928627:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533956.928628:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584533956.928629:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584533956.928631:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15756 00000100:00080000:22.0:1584533956.928634:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584533956.928641:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533956.928642:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533956.928643:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533956.928644:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533956.928645:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533956.928646:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533956.928647:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533956.928648:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533956.934606:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584533956.934614:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584533956.934616:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584533956.934617:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584533956.934619:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584533956.934621:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584533956.934623:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584533956.934625:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534037 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584533956.934653:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078405056 new x1661492078405120 00000800:00020000:4.0:1584533962.916092:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584533962.916096:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584533962.916098:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533962.916634:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584533962.928728:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533956/real 1584533956] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534043 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584533962.928750:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584533962.928752:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584533962.928754:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584533962.928763:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533962.928766:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533962.928767:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533962.928768:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533962.928769:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533962.928771:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584533962.928772:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584533962.928774:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15762 00000100:00080000:22.0:1584533962.928777:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584533962.928794:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533962.928796:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533962.928796:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533962.928798:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533962.928798:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533962.928800:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533962.928800:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533962.928802:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533962.935063:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584533962.935070:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584533962.935072:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584533962.935073:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584533962.935076:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584533962.935079:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584533962.935081:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584533962.935083:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534043 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584533962.935113:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078405120 new x1661492078405152 00000800:00000100:0.0:1584533969.141241:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533969.141628:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584533969.153683:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533962/real 1584533962] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534049 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584533969.153691:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584533969.153692:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584533969.153695:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584533969.153701:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533969.153703:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533969.153704:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533969.153705:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533969.153705:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533969.153707:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584533969.153708:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584533969.153709:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15768 00000100:00080000:22.0:1584533969.153711:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584533969.153720:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533969.153722:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533969.153722:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533969.153723:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533969.153724:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533969.153725:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533969.153726:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533969.153727:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533969.159650:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584533969.159654:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584533969.159656:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584533969.159657:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584533969.159658:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584533969.159660:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584533969.159662:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584533969.159664:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534049 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584533969.159672:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078405152 new x1661492078405184 00000800:00020000:0.0:1584533974.916065:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584533974.916068:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584533974.916070:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533974.916490:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584533974.928587:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533969/real 1584533969] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534056 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584533974.928610:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584533974.928611:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584533974.928613:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584533974.928618:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533974.928620:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533974.928630:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533974.928631:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533974.928632:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533974.928633:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584533974.928634:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584533974.928635:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15775 00000100:00080000:22.0:1584533974.928648:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584533974.928654:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533974.928655:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533974.928656:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533974.928657:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533974.928658:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533974.928659:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533974.928660:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533974.928661:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533974.934744:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584533974.934749:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584533974.934750:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584533974.934751:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584533974.934752:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584533974.934754:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584533974.934756:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584533974.934757:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534056 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584533974.934786:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078405184 new x1661492078405248 00000800:00020000:2.0:1584533980.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533980.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533980.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533980.916494:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584533980.928546:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533974/real 1584533974] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534061 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584533980.928554:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584533980.928555:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584533980.928556:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584533980.928569:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533980.928570:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533980.928571:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533980.928572:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533980.928573:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533980.928574:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584533980.928575:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584533980.928577:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15780 00000100:00080000:22.0:1584533980.928579:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584533980.928584:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533980.928585:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533980.928586:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533980.928587:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533980.928588:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533980.928589:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533980.928590:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533980.928591:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533980.957802:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584533980.957805:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584533980.957807:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584533980.957808:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584533980.957810:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584533980.957812:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584533980.957813:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584533980.957815:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534061 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584533980.957844:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078405248 new x1661492078405296 00000800:00020000:2.0:1584533986.916102:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533986.916105:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533986.916106:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533986.916562:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584533986.928679:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533980/real 1584533980] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534067 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584533986.928687:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584533986.928688:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584533986.928689:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584533986.928695:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533986.928697:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533986.928698:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533986.928699:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533986.928700:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533986.928701:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584533986.928702:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584533986.928703:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15786 00000100:00080000:22.0:1584533986.928705:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584533986.928714:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533986.928716:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533986.928716:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533986.928717:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533986.928719:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533986.928720:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533986.928721:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533986.928722:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533986.962648:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584533986.962656:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584533986.962659:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584533986.962661:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584533986.962664:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584533986.962668:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584533986.962672:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584533986.962676:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534067 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584533986.962724:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078405296 new x1661492078405344 00000800:00000100:0.0:1584533993.142297:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584533993.142787:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584533993.154844:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533986/real 1584533986] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534073 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584533993.154852:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584533993.154853:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584533993.154855:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584533993.154860:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533993.154862:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533993.154863:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533993.154864:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533993.154865:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533993.154866:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584533993.154866:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584533993.154868:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15792 00000100:00080000:22.0:1584533993.154870:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584533993.154878:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533993.154880:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533993.154881:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533993.154883:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533993.154883:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533993.154885:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533993.154885:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533993.154886:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533993.203872:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584533993.203875:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584533993.203877:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584533993.203877:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584533993.203879:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584533993.203881:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584533993.203882:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584533993.203884:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534073 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584533993.203892:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078405344 new x1661492078405376 00000800:00020000:2.0:1584533998.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584533998.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584533998.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584533998.916497:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584533998.928588:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533993/real 1584533993] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534080 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584533998.928607:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584533998.928608:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584533998.928609:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584533998.928615:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533998.928616:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533998.928624:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533998.928626:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533998.928626:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533998.928628:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584533998.928628:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584533998.928630:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15799 00000100:00080000:22.0:1584533998.928632:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584533998.928638:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533998.928639:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533998.928640:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533998.928641:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533998.928642:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533998.928643:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584533998.928644:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584533998.928645:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584533998.964908:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584533998.964911:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584533998.964913:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584533998.964914:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584533998.964916:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584533998.964918:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584533998.964919:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584533998.964921:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534080 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584533998.964950:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078405376 new x1661492078405424 00000800:00020000:2.0:1584534004.916082:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534004.916085:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534004.916088:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534004.916539:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584534004.928591:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584533998/real 1584533998] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534085 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584534004.928601:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584534004.928603:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584534004.928605:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584534004.928624:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534004.928627:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534004.928627:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534004.928629:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534004.928629:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534004.928631:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584534004.928632:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584534004.928634:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15804 00000100:00080000:22.0:1584534004.928638:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584534004.928645:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534004.928646:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534004.928647:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534004.928648:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534004.928649:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534004.928650:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534004.928651:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534004.928652:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534004.979051:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584534004.979058:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584534004.979060:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584534004.979061:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584534004.979063:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584534004.979065:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584534004.979067:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584534004.979069:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534085 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584534004.979099:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078405424 new x1661492078405488 00000800:00020000:2.0:1584534010.916104:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534010.916111:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534010.916116:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534010.916581:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584534010.928678:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534004/real 1584534004] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534091 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584534010.928701:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584534010.928703:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584534010.928704:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584534010.928713:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534010.928715:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534010.928716:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534010.928717:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534010.928718:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534010.928721:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584534010.928722:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584534010.928724:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15810 00000100:00080000:22.0:1584534010.928727:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584534010.928744:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534010.928745:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534010.928746:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534010.928747:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534010.928748:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534010.928749:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534010.928750:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534010.928751:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534010.979716:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584534010.979727:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584534010.979731:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584534010.979733:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584534010.979736:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584534010.979741:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584534010.979745:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584534010.979749:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534091 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584534010.979797:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078405488 new x1661492078405520 00000800:00000100:0.0:1584534017.141721:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534017.142183:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584534017.154246:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534010/real 1584534010] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534097 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584534017.154255:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584534017.154256:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584534017.154258:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584534017.154265:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534017.154267:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534017.154267:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534017.154272:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534017.154273:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534017.154274:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584534017.154275:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584534017.154277:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15816 00000100:00080000:22.0:1584534017.154281:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584534017.154295:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534017.154296:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534017.154297:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534017.154298:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534017.154298:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534017.154300:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534017.154311:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534017.154312:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534017.188038:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584534017.188045:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584534017.188047:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584534017.188048:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584534017.188051:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584534017.188053:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584534017.188055:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584534017.188057:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534097 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584534017.188091:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078405520 new x1661492078405568 00000800:00020000:4.0:1584534022.916100:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584534022.916106:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584534022.916110:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534022.916518:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584534022.928603:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534017/real 1584534017] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534104 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584534022.928622:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584534022.928623:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584534022.928624:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584534022.928630:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534022.928632:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534022.928642:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534022.928643:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534022.928644:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534022.928645:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584534022.928646:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584534022.928647:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15823 00000100:00080000:22.0:1584534022.928661:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584534022.928666:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534022.928667:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534022.928668:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534022.928669:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534022.928670:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534022.928671:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534022.928672:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534022.928673:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534022.982609:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584534022.982613:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584534022.982614:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584534022.982615:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584534022.982616:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584534022.982618:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584534022.982620:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584534022.982621:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534104 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584534022.982651:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078405568 new x1661492078405616 00000800:00020000:2.0:1584534028.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534028.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534028.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534028.916504:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584534028.928557:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534022/real 1584534022] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534109 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584534028.928566:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584534028.928567:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584534028.928569:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584534028.928584:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534028.928586:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534028.928587:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534028.928588:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534028.928589:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534028.928590:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584534028.928591:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584534028.928593:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15828 00000100:00080000:22.0:1584534028.928596:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584534028.928602:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534028.928603:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534028.928604:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534028.928605:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534028.928605:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534028.928607:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534028.928607:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534028.928609:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534028.974096:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584534028.974100:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584534028.974102:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584534028.974103:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584534028.974105:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584534028.974107:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584534028.974108:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584534028.974110:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534109 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584534028.974139:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078405616 new x1661492078405664 00000800:00020000:2.0:1584534034.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534034.916086:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534034.916090:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534034.916667:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584534034.928760:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534028/real 1584534028] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534115 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584534034.928783:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584534034.928784:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584534034.928785:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584534034.928791:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534034.928793:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534034.928794:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534034.928795:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534034.928796:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534034.928797:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584534034.928798:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584534034.928799:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15834 00000100:00080000:22.0:1584534034.928801:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584534034.928810:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534034.928811:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534034.928811:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534034.928813:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534034.928813:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534034.928814:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534034.928815:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534034.928816:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534034.992005:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584534034.992014:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584534034.992017:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584534034.992019:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584534034.992023:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584534034.992077:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584534034.992078:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584534034.992080:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534115 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584534034.992143:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078405664 new x1661492078405712 00000800:00000100:0.0:1584534041.142709:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534041.143141:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584534041.155211:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534034/real 1584534034] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534121 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584534041.155219:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584534041.155220:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584534041.155221:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584534041.155227:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534041.155229:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534041.155229:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534041.155230:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534041.155231:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534041.155233:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584534041.155233:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584534041.155235:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15840 00000100:00080000:22.0:1584534041.155237:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584534041.155245:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534041.155247:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534041.155247:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534041.155248:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534041.155249:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534041.155250:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534041.155252:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534041.155253:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534041.208859:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584534041.208863:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584534041.208864:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584534041.208865:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584534041.208866:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584534041.208868:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584534041.208869:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584534041.208871:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534121 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584534041.208900:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078405712 new x1661492078405744 00000800:00020000:2.0:1584534046.916100:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534046.916106:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534046.916110:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534046.916531:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584534046.928649:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534041/real 1584534041] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534128 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584534046.928657:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584534046.928658:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584534046.928659:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584534046.928665:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534046.928667:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534046.928676:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534046.928678:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534046.928679:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534046.928680:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584534046.928681:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584534046.928682:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15847 00000100:00080000:22.0:1584534046.928684:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584534046.928690:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534046.928691:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534046.928692:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534046.928693:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534046.928695:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534046.928696:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534046.928697:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534046.928698:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534046.978567:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584534046.978571:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584534046.978573:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584534046.978574:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584534046.978575:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584534046.978577:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584534046.978579:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584534046.978581:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534128 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584534046.978611:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078405744 new x1661492078405808 00000800:00020000:4.0:1584534052.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584534052.916081:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584534052.916083:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534052.916491:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584534052.928542:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534046/real 1584534046] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534133 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584534052.928550:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584534052.928551:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584534052.928552:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584534052.928566:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534052.928568:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534052.928568:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534052.928570:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534052.928570:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534052.928572:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584534052.928572:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584534052.928574:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15852 00000100:00080000:22.0:1584534052.928576:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584534052.928581:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534052.928582:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534052.928584:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534052.928585:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534052.928585:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534052.928587:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534052.928587:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534052.928588:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534052.985019:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584534052.985023:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584534052.985025:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584534052.985025:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584534052.985033:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584534052.985034:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584534052.985036:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584534052.985038:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534133 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584534052.985067:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078405808 new x1661492078405856 00000800:00020000:2.0:1584534058.916103:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534058.916109:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534058.916113:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534058.916588:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584534058.928680:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534052/real 1584534052] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534139 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584534058.928698:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584534058.928699:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584534058.928700:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584534058.928706:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534058.928707:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534058.928708:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534058.928710:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534058.928710:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534058.928711:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584534058.928712:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584534058.928714:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15858 00000100:00080000:22.0:1584534058.928715:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584534058.928724:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534058.928726:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534058.928726:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534058.928727:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534058.928728:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534058.928729:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534058.928730:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534058.928731:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534058.977307:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584534058.977311:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584534058.977313:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584534058.977313:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584534058.977315:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584534058.977317:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584534058.977318:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584534058.977320:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534139 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584534058.977354:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078405856 new x1661492078405888 00000800:00000100:0.0:1584534065.142574:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534065.143021:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584534065.155106:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534058/real 1584534058] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534145 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584534065.155116:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584534065.155117:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584534065.155119:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584534065.155126:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534065.155128:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534065.155129:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534065.155130:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534065.155131:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534065.155133:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584534065.155134:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584534065.155137:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15864 00000100:00080000:22.0:1584534065.155139:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584534065.155150:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534065.155151:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534065.155152:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534065.155154:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534065.155154:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534065.155156:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534065.155156:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534065.155157:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534065.214582:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584534065.214588:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584534065.214589:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584534065.214591:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584534065.214592:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584534065.214595:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584534065.214596:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584534065.214598:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534145 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584534065.214628:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078405888 new x1661492078405936 00000800:00020000:4.0:1584534071.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584534071.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584534071.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534071.916526:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584534071.928646:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534065/real 1584534065] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534152 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584534071.928656:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584534071.928657:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584534071.928659:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584534071.928666:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534071.928668:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534071.928680:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534071.928681:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534071.928682:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534071.928685:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584534071.928686:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584534071.928688:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15871 00000100:00080000:22.0:1584534071.928691:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584534071.928698:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534071.928700:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534071.928700:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534071.928702:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534071.928703:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534071.928704:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534071.928705:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534071.928706:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534071.976967:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584534071.976973:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584534071.976976:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584534071.976977:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584534071.976979:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584534071.976981:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584534071.976982:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584534071.976984:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534152 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584534071.977014:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078405936 new x1661492078405984 00000800:00000100:0.0:1584534078.140789:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534078.141232:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584534078.153285:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534071/real 1584534071] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534158 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584534078.153295:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584534078.153297:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584534078.153299:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584534078.153318:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534078.153320:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534078.153321:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534078.153323:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534078.153324:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534078.153326:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584534078.153326:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584534078.153328:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15877 00000100:00080000:22.0:1584534078.153331:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584534078.153339:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534078.153340:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534078.153341:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534078.153342:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534078.153343:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534078.153344:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534078.153345:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584534078.153346:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534078.214183:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584534078.214189:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584534078.214192:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584534078.214193:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584534078.214195:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584534078.214197:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584534078.214199:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584534078.214201:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534158 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584534078.214211:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078405984 new x1661492078406048 00000800:00000100:0.0:1584534084.141057:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534084.141506:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584534084.153581:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534078/real 1584534078] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534165 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584534084.153592:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584534084.153594:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584534084.153596:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584534084.153604:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584534084.153607:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584534084.153607:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584534084.153609:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584534084.153610:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584534084.153611:0:4532:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584534084.153612:0:4532:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584534084.153615:0:4532:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15884 00000100:00080000:2.0:1584534084.153618:0:4532:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584534084.153637:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584534084.153638:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584534084.153639:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584534084.153640:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584534084.153641:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584534084.153642:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584534084.153643:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584534084.153644:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534084.196179:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584534084.196186:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584534084.196188:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584534084.196190:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584534084.196192:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584534084.196195:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584534084.196196:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584534084.196198:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534165 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584534084.196208:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078406048 new x1661492078406080 00000800:00020000:2.0:1584534090.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534090.916081:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534090.916083:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534090.916465:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584534090.928542:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534084/real 1584534084] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534171 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584534090.928551:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584534090.928552:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584534090.928553:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584534090.928560:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584534090.928562:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584534090.928562:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584534090.928564:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584534090.928564:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584534090.928566:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584534090.928567:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584534090.928569:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15890 00000100:00080000:4.0:1584534090.928571:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584534090.928583:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584534090.928585:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584534090.928585:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584534090.928587:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584534090.928587:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584534090.928589:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584534090.928589:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584534090.928590:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534090.984439:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584534090.984443:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584534090.984445:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584534090.984446:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584534090.984448:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584534090.984450:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584534090.984451:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584534090.984453:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534171 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584534090.984482:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078406080 new x1661492078406112 00000800:00000100:0.0:1584534097.141603:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534097.142063:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584534097.154133:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534090/real 1584534090] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534177 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584534097.154141:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584534097.154142:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584534097.154143:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584534097.154149:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584534097.154150:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584534097.154159:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584534097.154161:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584534097.154162:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584534097.154164:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584534097.154164:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584534097.154166:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15896 00000100:00080000:4.0:1584534097.154168:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584534097.154173:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584534097.154175:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584534097.154175:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584534097.154177:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584534097.154177:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584534097.154178:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584534097.154179:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584534097.154180:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534097.201246:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584534097.201250:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584534097.201252:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584534097.201253:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584534097.201254:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584534097.201256:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584534097.201258:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584534097.201259:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534177 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534097.201290:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078406112 new x1661492078406176 00000800:00000100:0.0:1584534103.142781:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534103.143187:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534103.155235:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534097/real 1584534097] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534184 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534103.155242:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534103.155243:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534103.155245:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534103.155257:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534103.155259:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534103.155260:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534103.155261:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534103.155261:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534103.155263:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534103.155263:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534103.155265:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15903 00000100:00080000:8.0:1584534103.155267:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534103.155272:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534103.155273:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534103.155275:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534103.155277:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534103.155277:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534103.155278:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534103.155279:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534103.155280:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534103.187288:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584534103.187292:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584534103.187293:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584534103.187294:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584534103.187295:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584534103.187297:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584534103.187299:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584534103.187300:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534184 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534103.187329:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078406176 new x1661492078406224 00000800:00000100:0.0:1584534109.142453:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534109.142879:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534109.154954:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534103/real 1584534103] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534190 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534109.154962:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534109.154963:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534109.154964:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534109.154971:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534109.154972:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534109.154973:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534109.154974:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534109.154975:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534109.154977:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534109.154977:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534109.154979:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15909 00000100:00080000:8.0:1584534109.154981:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534109.154991:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534109.154993:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534109.154993:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534109.154995:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534109.154995:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534109.154996:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534109.154997:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534109.154998:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534109.197231:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584534109.197235:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584534109.197237:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584534109.197238:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584534109.197239:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584534109.197241:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584534109.197243:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584534109.197244:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534190 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534109.197273:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078406224 new x1661492078406272 00000800:00000100:0.0:1584534115.142223:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534115.142683:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534115.154751:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534109/real 1584534109] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534196 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534115.154758:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534115.154759:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534115.154761:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534115.154766:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534115.154768:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534115.154769:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534115.154770:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534115.154771:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534115.154772:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534115.154773:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534115.154775:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15915 00000100:00080000:8.0:1584534115.154777:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534115.154785:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534115.154787:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534115.154787:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534115.154789:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534115.154789:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534115.154790:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534115.154791:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534115.154792:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534115.205787:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584534115.205791:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584534115.205793:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584534115.205794:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584534115.205795:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584534115.205797:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584534115.205798:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584534115.205800:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534196 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534115.205830:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078406272 new x1661492078406304 00000800:00000100:0.0:1584534121.142478:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534121.142913:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534121.154981:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534115/real 1584534115] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534202 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534121.154989:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534121.154990:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534121.154991:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534121.154997:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534121.154999:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534121.155008:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534121.155009:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534121.155010:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534121.155012:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534121.155013:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534121.155015:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15921 00000100:00080000:8.0:1584534121.155017:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534121.155022:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534121.155023:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534121.155024:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534121.155025:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534121.155026:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534121.155085:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534121.155086:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534121.155087:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534121.216789:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584534121.216793:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584534121.216795:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584534121.216796:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584534121.216797:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584534121.216799:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584534121.216800:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584534121.216802:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534202 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534121.216831:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078406304 new x1661492078406352 00000800:00000100:0.0:1584534127.142647:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534127.143140:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534127.155199:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534121/real 1584534121] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534208 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534127.155209:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534127.155211:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534127.155213:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534127.155233:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534127.155235:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534127.155236:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534127.155238:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534127.155239:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534127.155241:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534127.155242:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534127.155245:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15927 00000100:00080000:8.0:1584534127.155248:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534127.155257:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534127.155258:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534127.155259:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534127.155260:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534127.155261:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534127.155262:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534127.155263:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534127.155264:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534127.185444:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534127.185451:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534127.185454:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534127.185455:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534127.185457:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534127.185460:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534127.185461:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534127.185463:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534208 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534127.185495:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078406352 new x1661492078406416 00000800:00000100:0.0:1584534133.140938:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534133.141488:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534133.153565:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534127/real 1584534127] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534214 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534133.153575:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534133.153577:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534133.153579:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534133.153587:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534133.153590:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534133.153591:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534133.153592:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534133.153593:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534133.153595:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534133.153596:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534133.153598:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15933 00000100:00080000:8.0:1584534133.153601:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534133.153618:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534133.153619:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534133.153620:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534133.153621:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534133.153622:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534133.153623:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534133.153624:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534133.153625:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534133.205016:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534133.205023:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534133.205025:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534133.205038:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534133.205041:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534133.205043:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534133.205045:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534133.205047:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534214 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534133.205077:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078406416 new x1661492078406448 00000800:00000100:0.0:1584534139.139574:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534139.140045:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534139.152117:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534133/real 1584534133] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534220 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534139.195905:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534139.195906:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534139.195908:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534139.195914:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534139.195916:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534139.195916:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534139.195917:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534139.195918:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534139.195920:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534139.195920:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534139.195922:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15939 00000100:00080000:8.0:1584534139.195924:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534139.195935:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534139.195937:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534139.195937:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534139.195939:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534139.195939:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534139.195940:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534139.195941:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534139.195942:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534139.240048:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534139.240054:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534139.240056:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534139.240057:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534139.240058:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534139.240061:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534139.240063:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534139.240065:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534220 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534139.240093:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078406448 new x1661492078406496 00000800:00000100:0.0:1584534145.140980:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534145.141389:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534145.153438:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534139/real 1584534139] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534226 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534145.153450:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534145.153451:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534145.153452:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534145.153457:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534145.153459:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534145.153468:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534145.153469:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534145.153470:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534145.153471:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534145.153472:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534145.153473:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15945 00000100:00080000:8.0:1584534145.153475:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534145.153481:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534145.153482:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534145.153482:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534145.153484:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534145.153484:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534145.153485:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534145.153486:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534145.153487:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534145.182884:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534145.182888:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534145.182890:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534145.182890:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534145.182892:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534145.182894:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534145.182895:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534145.182897:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534226 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534145.182927:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078406496 new x1661492078406544 00000800:00000100:0.0:1584534151.140090:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534151.140472:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534151.152521:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534145/real 1584534145] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534232 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534151.152529:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534151.152530:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534151.152531:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534151.152545:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534151.152546:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534151.152547:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534151.152548:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534151.152549:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534151.152550:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534151.152551:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534151.152552:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15951 00000100:00080000:8.0:1584534151.152554:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534151.152560:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534151.152561:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534151.152562:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534151.152563:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534151.152563:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534151.152565:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534151.152566:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534151.152567:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534151.205968:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534151.205972:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534151.205974:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534151.205975:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534151.205976:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534151.205978:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534151.205979:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534151.205981:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534232 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534151.206010:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078406544 new x1661492078406592 00000800:00000100:0.0:1584534157.140203:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534157.140616:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534157.152666:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534151/real 1584534151] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534238 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534157.152673:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534157.152675:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534157.152676:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534157.152681:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534157.152682:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534157.152683:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534157.152685:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534157.152685:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534157.152687:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534157.152687:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534157.152689:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15957 00000100:00080000:8.0:1584534157.152691:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534157.152699:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534157.152700:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534157.152701:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534157.152702:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534157.152704:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534157.152705:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534157.152706:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534157.152707:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534157.210538:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534157.210542:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534157.210544:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534157.210544:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534157.210546:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534157.210548:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534157.210550:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534157.210551:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534238 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534157.210559:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078406592 new x1661492078406640 00000800:00000100:0.0:1584534163.140473:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534163.141088:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534163.153140:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534157/real 1584534157] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534244 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534163.153147:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534163.153148:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534163.153149:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534163.153154:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534163.153156:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534163.153157:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534163.153158:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534163.153158:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534163.153160:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534163.153160:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534163.153162:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15963 00000100:00080000:8.0:1584534163.153164:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534163.153172:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534163.153173:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534163.153175:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534163.153176:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534163.153177:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534163.153178:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534163.153178:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534163.153180:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534163.207040:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534163.207044:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534163.207045:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534163.207046:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534163.207048:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534163.207049:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534163.207051:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534163.207053:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534244 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534163.207085:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078406640 new x1661492078406672 00000800:00000100:0.0:1584534169.141726:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534169.142110:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534169.154160:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534163/real 1584534163] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534250 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534169.154168:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534169.154169:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534169.154170:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534169.154175:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534169.154177:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534169.154185:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534169.154187:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534169.154187:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534169.154189:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534169.154190:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534169.154191:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15969 00000100:00080000:8.0:1584534169.154193:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534169.154200:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534169.154201:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534169.154202:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534169.154203:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534169.154203:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534169.154205:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534169.154205:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534169.154206:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534169.181337:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534169.181342:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534169.181343:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534169.181344:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534169.181345:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534169.181347:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534169.181349:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534169.181350:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534250 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534169.181379:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078406672 new x1661492078406736 00000800:00000100:0.0:1584534175.141348:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534175.141762:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534175.153832:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534169/real 1584534169] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534256 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534175.153840:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534175.153841:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534175.153842:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534175.153856:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534175.153858:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534175.153858:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534175.153860:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534175.153860:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534175.153862:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534175.153862:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534175.153865:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15975 00000100:00080000:8.0:1584534175.153867:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534175.153872:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534175.153873:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534175.153874:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534175.153875:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534175.153876:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534175.153877:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534175.153877:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534175.153878:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534175.230730:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534175.230734:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534175.230735:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534175.230736:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534175.230737:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534175.230739:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534175.230741:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534175.230742:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534256 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534175.230771:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078406736 new x1661492078406784 00000800:00000100:0.0:1584534181.141280:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534181.141694:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534181.153742:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534175/real 1584534175] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534262 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534181.153750:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534181.153751:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534181.153753:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534181.153757:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534181.153759:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534181.153760:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534181.153761:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534181.153762:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534181.153764:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534181.153765:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534181.153766:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15981 00000100:00080000:8.0:1584534181.153768:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534181.153777:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534181.153778:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534181.153778:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534181.153780:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534181.153780:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534181.153781:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534181.153782:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534181.153783:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534181.210436:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534181.210440:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534181.210441:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534181.210442:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534181.210444:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534181.210445:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534181.210447:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534181.210449:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534262 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534181.210478:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078406784 new x1661492078406816 00000800:00020000:4.0:1584534187.916065:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584534187.916069:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584534187.916071:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534187.916575:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534187.928670:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534181/real 1584534181] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534268 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534187.928680:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534187.928682:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534187.928684:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534187.928692:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534187.928694:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534187.928694:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534187.928697:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534187.928697:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534187.928699:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534187.928700:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534187.928702:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15987 00000100:00080000:8.0:1584534187.928705:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534187.928719:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534187.928720:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534187.928721:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534187.928723:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534187.928723:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534187.928724:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534187.928725:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534187.928726:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534187.971648:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534187.971655:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534187.971658:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534187.971659:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534187.971661:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534187.971664:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534187.971665:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534187.971667:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534268 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534187.971677:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078406816 new x1661492078406864 00000800:00000100:0.0:1584534194.141384:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534194.141777:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534194.153829:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534187/real 1584534187] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534274 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534194.153838:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534194.153839:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534194.153841:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534194.153848:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534194.153851:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534194.153864:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534194.153865:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534194.153866:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534194.153868:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534194.153869:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534194.153871:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15993 00000100:00080000:8.0:1584534194.153873:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534194.153880:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534194.153882:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534194.153883:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534194.153884:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534194.153885:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534194.153886:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534194.153887:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534194.153888:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534194.209070:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534194.209076:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534194.209079:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534194.209080:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534194.209082:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534194.209084:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534194.209086:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534194.209088:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534274 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534194.209098:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078406864 new x1661492078406912 00000800:00000100:0.0:1584534200.142687:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534200.143108:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534200.155158:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534194/real 1584534194] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534200.155166:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534200.155167:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534200.155170:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534200.155184:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534200.155185:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534200.155186:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534200.155187:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534200.155188:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534200.155190:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534200.155190:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534200.155192:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16000 00000100:00080000:8.0:1584534200.155194:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534200.155200:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534200.155202:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534200.155203:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534200.155204:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534200.155204:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534200.155205:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534200.155206:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534200.155208:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534200.221521:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534200.221527:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534200.221529:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534200.221530:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534200.221531:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534200.221533:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534200.221535:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534200.221536:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534200.221546:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078406912 new x1661492078406976 00000800:00020000:2.0:1584534206.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534206.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534206.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534206.916564:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534206.928649:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534200/real 1584534200] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534206.928672:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534206.928673:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534206.928675:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534206.928680:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534206.928682:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534206.928683:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534206.928684:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534206.928684:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534206.928686:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534206.928686:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534206.928688:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16006 00000100:00080000:8.0:1584534206.928690:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534206.928698:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534206.928699:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534206.928700:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534206.928701:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534206.928702:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534206.928703:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534206.928703:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534206.928705:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534206.978345:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534206.978349:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534206.978351:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534206.978352:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534206.978353:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534206.978355:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534206.978357:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534206.978358:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534206.978388:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078406976 new x1661492078407008 00000800:00000100:0.0:1584534213.142231:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534213.142758:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534213.154807:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534206/real 1584534206] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534213.154814:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534213.180161:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534213.180163:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534213.180167:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534213.180169:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534213.180169:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534213.180170:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534213.180171:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534213.180173:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534213.180173:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534213.180175:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16012 00000100:00080000:8.0:1584534213.180177:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534213.180185:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534213.180186:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534213.180187:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534213.180188:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534213.180188:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534213.180189:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534213.180191:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534213.180192:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534213.245183:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534213.245187:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534213.245189:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534213.245190:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534213.245191:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534213.245193:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534213.245195:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534213.245197:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534213.245226:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078407008 new x1661492078407056 00000800:00000100:0.0:1584534219.142611:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534219.143033:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534219.155097:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534213/real 1584534213] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534219.155104:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534219.155105:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534219.155107:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534219.155111:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534219.155113:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534219.155121:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534219.155122:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534219.155122:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534219.155124:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534219.155124:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534219.155126:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16019 00000100:00080000:8.0:1584534219.155128:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534219.155133:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534219.155134:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534219.155135:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534219.155136:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534219.155137:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534219.155139:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534219.155139:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534219.155140:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534219.207527:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534219.207531:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534219.207533:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534219.207534:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534219.207535:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534219.207537:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534219.207538:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534219.225333:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534219.225345:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078407056 new x1661492078407104 00000800:00020000:2.0:1584534225.916073:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534225.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534225.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534225.916484:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534225.928560:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534219/real 1584534219] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534225.928568:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534225.928569:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534225.928570:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534225.928582:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534225.928584:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534225.928585:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534225.928586:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534225.928587:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534225.928588:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534225.928589:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534225.928590:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16025 00000100:00080000:8.0:1584534225.928592:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534225.928598:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534225.928599:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534225.928601:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534225.928602:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534225.928602:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534225.928603:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534225.928604:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534225.928605:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534225.994051:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534225.994055:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534225.994057:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534225.994058:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534225.994059:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534225.994061:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534225.994063:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534225.994064:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534225.994093:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078407104 new x1661492078407152 00000800:00000100:0.0:1584534232.142959:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534232.143491:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534232.155561:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534225/real 1584534225] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534232.155569:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534232.155570:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534232.155572:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534232.155577:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534232.155579:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534232.155580:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534232.155581:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534232.155582:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534232.155583:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534232.155584:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534232.155585:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16031 00000100:00080000:8.0:1584534232.155587:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534232.155598:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534232.155599:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534232.155600:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534232.155601:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534232.155602:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534232.155603:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534232.155604:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534232.155605:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534232.206737:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534232.206742:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534232.206743:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534232.206744:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534232.206745:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534232.206748:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534232.206749:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534232.206751:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534232.206781:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078407152 new x1661492078407200 00000800:00000100:0.0:1584534238.143319:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534238.143704:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534238.155764:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534232/real 1584534232] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534319 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534238.155772:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534238.155773:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534238.155774:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534238.155779:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534238.155781:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534238.155781:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534238.155783:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534238.155783:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534238.155785:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534238.155785:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534238.155788:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16038 00000100:00080000:8.0:1584534238.155790:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534238.155798:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534238.155799:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534238.155800:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534238.155801:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534238.155802:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534238.155803:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534238.155803:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534238.155804:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534238.208338:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534238.208342:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534238.208344:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534238.208344:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534238.208346:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534238.208347:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534238.208349:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534238.208351:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534319 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534238.208380:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078407200 new x1661492078407232 00000800:00020000:2.0:1584534244.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534244.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534244.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534244.916507:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534244.928585:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534238/real 1584534238] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534325 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534244.928594:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534244.928595:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534244.928596:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534244.928602:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534244.928604:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534244.928611:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534244.928612:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534244.928613:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534244.928615:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534244.928616:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534244.928618:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16044 00000100:00080000:8.0:1584534244.928620:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534244.928626:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534244.928627:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534244.928628:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534244.928629:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534244.928630:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534244.928631:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534244.928631:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534244.928633:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534244.970099:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534244.970104:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534244.970105:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534244.970106:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534244.970107:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534244.970109:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534244.970111:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534244.970113:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534325 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534244.970122:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078407232 new x1661492078407296 00000800:00000100:0.0:1584534251.143782:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534251.144242:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534251.156311:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534244/real 1584534244] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534331 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534251.156321:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534251.156322:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534251.156324:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534251.156343:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534251.156345:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534251.156346:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534251.156359:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534251.156360:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534251.156361:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534251.156362:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534251.156364:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16050 00000100:00080000:8.0:1584534251.156367:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534251.156385:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534251.156386:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534251.156387:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534251.156388:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534251.156389:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534251.156390:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534251.156391:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534251.156403:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534251.216674:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534251.216680:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534251.216683:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534251.216684:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534251.216686:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534251.216688:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534251.216690:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534251.216692:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534331 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534251.216722:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078407296 new x1661492078407344 00000800:00000100:0.0:1584534257.142426:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534257.143014:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534257.155079:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534251/real 1584534251] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534338 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534257.155089:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534257.155090:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534257.155092:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534257.155100:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534257.155102:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534257.155103:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534257.155104:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534257.155105:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534257.155107:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534257.155108:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534257.155109:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16057 00000100:00080000:8.0:1584534257.155112:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534257.155126:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534257.155127:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534257.155128:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534257.155129:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534257.155129:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534257.155131:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534257.155142:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534257.155143:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534257.217748:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534257.217760:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534257.217763:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534257.217766:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534257.217769:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534257.217774:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534257.217778:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534257.217782:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534338 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534257.217832:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078407344 new x1661492078407392 00000800:00020000:2.0:1584534263.916101:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534263.916103:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534263.916106:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534263.916576:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534263.928630:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534257/real 1584534257] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534344 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534263.928638:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534263.928639:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534263.928641:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534263.928648:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534263.928650:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534263.928651:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534263.928652:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534263.928653:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534263.928655:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534263.928655:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534263.928657:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16063 00000100:00080000:8.0:1584534263.928659:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534263.928673:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534263.928674:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534263.928675:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534263.928676:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534263.928677:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534263.928678:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534263.928679:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534263.928680:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534263.934547:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534263.934553:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534263.934555:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534263.934556:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534263.934558:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534263.934560:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534263.934562:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534263.934563:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534344 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534263.934593:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078407392 new x1661492078407424 00000800:00000100:0.0:1584534270.140483:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534270.140940:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534270.153008:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534263/real 1584534263] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534350 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534270.153020:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534270.153022:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534270.153023:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534270.153032:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534270.153034:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534270.153043:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534270.153044:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534270.153045:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534270.153047:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534270.153047:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534270.153049:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16069 00000100:00080000:8.0:1584534270.153051:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534270.153056:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534270.153058:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534270.153058:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534270.153059:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534270.153060:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534270.153061:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534270.153062:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534270.153063:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534270.159339:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534270.159344:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534270.159345:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534270.159346:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534270.159348:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534270.159349:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534270.159351:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534270.159353:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534350 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534270.159381:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078407424 new x1661492078407472 00000800:00000100:0.0:1584534276.140695:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534276.141120:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534276.153188:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534270/real 1584534270] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534357 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534276.153195:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534276.153196:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534276.153198:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534276.153211:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534276.153213:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534276.153214:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534276.153215:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534276.153216:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534276.153217:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534276.153218:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534276.153219:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16076 00000100:00080000:8.0:1584534276.153222:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534276.153227:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534276.153229:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534276.153229:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534276.153230:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534276.153231:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534276.153232:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534276.153234:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534276.153235:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534276.159172:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534276.159176:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534276.159178:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534276.159179:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534276.159180:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534276.159182:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534276.159183:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534276.159185:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534357 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534276.159220:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078407472 new x1661492078407536 00000800:00000100:0.0:1584534282.140951:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534282.141436:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534282.153506:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534276/real 1584534276] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534363 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534282.153514:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534282.153515:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534282.153516:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534282.153522:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534282.153524:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534282.153524:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534282.153526:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534282.153526:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534282.153528:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534282.153528:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534282.153530:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16082 00000100:00080000:8.0:1584534282.153532:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534282.153540:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534282.153541:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534282.153542:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534282.153543:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534282.153546:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534282.153548:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534282.153548:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534282.153549:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534282.159287:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534282.159291:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534282.159293:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534282.159293:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534282.159295:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534282.159297:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534282.159298:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534282.159300:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534363 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534282.159328:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078407536 new x1661492078407568 00000800:00020000:4.0:1584534288.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584534288.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584534288.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534288.916492:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534288.928568:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534282/real 1584534282] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534369 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534288.928576:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534288.928577:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534288.928579:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534288.928585:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534288.928586:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534288.928587:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534288.928588:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534288.928589:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534288.928590:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534288.928591:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534288.928592:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16088 00000100:00080000:8.0:1584534288.928594:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534288.928604:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534288.928605:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534288.928607:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534288.928608:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534288.928608:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534288.928610:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534288.928610:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534288.928611:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534288.934579:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534288.934584:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534288.934586:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534288.934587:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534288.934588:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534288.934590:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534288.934592:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534288.934593:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534369 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534288.934622:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078407568 new x1661492078407616 00000800:00000100:0.0:1584534295.141423:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534295.141861:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534295.153918:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534288/real 1584534288] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534375 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534295.153925:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534295.153925:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534295.153927:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534295.153932:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534295.153933:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534295.153941:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534295.153942:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534295.153942:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534295.153944:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534295.153944:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534295.153956:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16094 00000100:00080000:8.0:1584534295.153958:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534295.153964:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534295.153965:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534295.153977:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534295.153978:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534295.153979:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534295.153980:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534295.153980:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534295.153981:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534295.159939:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534295.159943:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534295.159945:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534295.159945:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534295.159947:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534295.159948:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534295.159950:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534295.159952:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534375 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534295.159982:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078407616 new x1661492078407664 00000800:00000100:0.0:1584534301.141681:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534301.142108:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534301.154174:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534295/real 1584534295] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534382 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534301.154182:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534301.154183:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534301.154184:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534301.154198:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534301.154200:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534301.154200:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534301.154202:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534301.154202:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534301.154203:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534301.154204:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534301.154207:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16101 00000100:00080000:8.0:1584534301.154209:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534301.154214:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534301.154215:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534301.154216:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534301.154217:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534301.154217:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534301.154218:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534301.154219:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534301.154220:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534301.160022:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534301.160026:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534301.160058:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534301.160058:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534301.160060:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534301.160062:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534301.160063:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534301.160065:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534382 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534301.160094:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078407664 new x1661492078407712 00000800:00020000:2.0:1584534307.916064:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534307.916067:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534307.916069:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534307.916555:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534307.928632:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534301/real 1584534301] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534388 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534307.928642:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534307.928644:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534307.928646:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534307.928653:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534307.928655:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534307.928656:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534307.928657:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534307.928658:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534307.928661:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534307.928662:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534307.928664:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16107 00000100:00080000:8.0:1584534307.928667:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534307.928680:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534307.928682:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534307.928682:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534307.928684:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534307.928684:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534307.928686:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534307.928686:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534307.928687:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534307.934498:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534307.934504:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534307.934506:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534307.934508:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534307.934509:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534307.934511:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534307.934513:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534307.934515:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534388 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534307.934544:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078407712 new x1661492078407760 00000800:00000100:0.0:1584534314.141545:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534314.141964:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534314.154039:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534307/real 1584534307] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534394 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534314.154050:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534314.154052:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534314.154054:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534314.154062:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534314.154064:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534314.154065:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534314.154067:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534314.154068:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534314.154070:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534314.154071:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534314.154073:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16113 00000100:00080000:8.0:1584534314.154076:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534314.154104:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534314.154105:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534314.154106:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534314.154107:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534314.154108:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534314.154109:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534314.154110:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534314.154111:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534314.160005:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534314.160013:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534314.160015:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534314.160016:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534314.160018:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534314.160020:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534314.160022:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534314.160024:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534394 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534314.160064:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078407760 new x1661492078407792 00000800:00000100:0.0:1584534320.141097:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534320.141587:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534320.153668:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534314/real 1584534314] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534401 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534320.153676:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534320.153678:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534320.153679:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534320.153686:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534320.153688:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534320.153699:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534320.153700:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534320.153701:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534320.153703:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534320.153704:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534320.153705:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16120 00000100:00080000:8.0:1584534320.153708:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534320.153714:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534320.153715:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534320.153716:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534320.153717:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534320.153718:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534320.153719:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534320.153720:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534320.153722:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534320.159520:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534320.159525:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534320.159527:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534320.159528:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534320.159530:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534320.159532:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534320.159534:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534320.159536:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534401 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534320.159565:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078407792 new x1661492078407856 00000800:00020000:2.0:1584534326.916067:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534326.916069:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534326.916072:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534326.916548:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534326.928602:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534320/real 1584534320] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534407 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534326.928610:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534326.928612:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534326.928614:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534326.928631:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534326.928633:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534326.928633:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534326.928635:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534326.928635:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534326.928637:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534326.928638:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534326.928639:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16126 00000100:00080000:8.0:1584534326.928642:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534326.928649:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534326.928650:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534326.928651:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534326.928652:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534326.928653:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534326.928654:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534326.928655:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534326.928656:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534326.934571:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534326.934577:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534326.934579:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534326.934580:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534326.934582:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534326.934584:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534326.934586:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534326.934588:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534407 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534326.934616:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078407856 new x1661492078407904 00000800:00000100:0.0:1584534333.141583:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534333.142121:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534333.154191:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534326/real 1584534326] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534413 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534333.154204:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534333.154205:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534333.154206:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534333.154212:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534333.154214:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534333.154214:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534333.154216:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534333.154216:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534333.154218:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534333.154218:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534333.154220:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16132 00000100:00080000:8.0:1584534333.154222:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534333.154231:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534333.154233:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534333.154233:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534333.154234:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534333.154235:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534333.154236:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534333.154237:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534333.154238:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534333.160012:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534333.160017:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534333.160019:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534333.160019:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534333.160021:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534333.160023:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534333.160024:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534333.160041:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534413 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534333.160071:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078407904 new x1661492078407952 00000800:00000100:0.0:1584534339.141886:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534339.142320:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534339.154396:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534333/real 1584534333] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534420 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534339.154404:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534339.154405:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534339.154406:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534339.154412:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534339.154413:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534339.154414:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534339.154415:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534339.154416:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534339.154417:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534339.154418:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534339.154419:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16139 00000100:00080000:8.0:1584534339.154421:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534339.154430:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534339.154431:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534339.154431:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534339.154433:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534339.154433:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534339.154434:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534339.154436:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534339.154437:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534339.160371:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534339.160375:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534339.160377:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534339.160377:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534339.160379:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534339.160381:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534339.160382:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534339.160384:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534420 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534339.160413:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078407952 new x1661492078407984 00000800:00020000:0.0:1584534345.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584534345.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584534345.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534345.916490:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534345.928538:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534339/real 1584534339] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534426 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534345.928546:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534345.928547:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534345.928549:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534345.928554:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534345.928556:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534345.928565:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534345.928566:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534345.928566:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534345.928568:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534345.928568:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534345.928570:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16145 00000100:00080000:8.0:1584534345.928572:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534345.928577:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534345.928578:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534345.928579:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534345.928580:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534345.928583:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534345.928584:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534345.928585:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534345.928586:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534345.934464:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534345.934468:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534345.934470:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534345.934471:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534345.934472:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534345.934474:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534345.934475:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534345.934477:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534426 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534345.934507:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078407984 new x1661492078408032 00000800:00000100:0.0:1584534352.142378:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534352.142810:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534352.154870:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534345/real 1584534345] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534432 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534352.154878:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534352.154879:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534352.154880:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534352.154895:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534352.154897:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534352.154898:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534352.154899:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534352.154899:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534352.154901:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534352.154901:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534352.154903:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16151 00000100:00080000:8.0:1584534352.154905:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534352.154910:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534352.154912:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534352.154913:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534352.154914:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534352.154915:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534352.154916:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534352.154917:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534352.154918:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534352.160731:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534352.160736:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534352.160737:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534352.160738:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534352.160739:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534352.160741:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534352.160743:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534352.160745:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534432 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534352.160774:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078408032 new x1661492078408096 00000800:00000100:0.0:1584534358.142705:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534358.143166:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534358.155217:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534352/real 1584534352] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534439 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534358.155225:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534358.155226:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534358.155227:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534358.155232:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534358.155234:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534358.155234:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534358.155236:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534358.155236:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534358.155237:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534358.155238:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534358.155240:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16158 00000100:00080000:8.0:1584534358.155242:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534358.155251:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534358.155252:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534358.155253:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534358.155254:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534358.155255:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534358.155256:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534358.155257:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534358.155258:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534358.161194:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534358.161198:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534358.161200:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534358.161200:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534358.161202:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534358.161203:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534358.161205:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534358.161207:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534439 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534358.161235:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078408096 new x1661492078408128 00000800:00020000:2.0:1584534364.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534364.938895:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534364.963072:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534364.963553:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534364.975611:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534358/real 1584534358] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534445 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534364.975621:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534364.975623:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534364.975625:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534364.975632:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534364.975634:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534364.975635:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534364.975637:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534364.975637:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534364.975639:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534364.975640:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534364.975643:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16164 00000100:00080000:8.0:1584534364.975646:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534364.975660:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534364.975662:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534364.975662:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534364.975664:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534364.975664:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534364.975666:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534364.975666:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534364.975668:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534364.981525:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534364.981531:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534364.981534:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534364.981535:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534364.981537:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534364.981539:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534364.981541:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534364.981543:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534445 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534364.981576:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078408128 new x1661492078408176 00000800:00000100:0.0:1584534371.141530:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534371.141945:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534371.153993:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534364/real 1584534364] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534451 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534371.154004:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534371.154006:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534371.154008:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534371.154015:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534371.154017:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534371.154051:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534371.154052:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534371.154053:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534371.154056:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534371.154057:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534371.154059:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16170 00000100:00080000:8.0:1584534371.154062:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534371.154071:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534371.154072:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534371.154073:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534371.154075:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534371.154076:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534371.154077:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534371.154077:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534371.154079:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534371.159966:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534371.159974:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534371.159976:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534371.159977:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534371.159979:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534371.159982:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534371.159983:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534371.159985:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534451 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584534371.160016:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078408176 new x1661492078408224 00000800:00000100:0.0:1584534377.140681:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534377.141166:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584534377.153239:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534371/real 1584534371] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534458 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584534377.153246:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584534377.153247:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584534377.153249:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584534377.153264:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534377.153265:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534377.153266:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534377.153268:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534377.153269:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534377.153281:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584534377.153281:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584534377.153283:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16177 00000100:00080000:8.0:1584534377.153286:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584534377.153292:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534377.153293:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534377.153294:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534377.153295:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534377.153295:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534377.153297:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534377.153297:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584534377.153299:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534377.159526:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534377.159531:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534377.159533:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534377.159534:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534377.159535:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534377.159537:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534377.159539:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534377.159541:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534458 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534377.159575:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078408224 new x1661492078408288 00000800:00000100:0.0:1584534383.141009:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534383.141446:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534383.153497:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534377/real 1584534377] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534464 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534383.153505:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534383.153506:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534383.153507:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534383.153513:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534383.153515:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534383.153515:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534383.153516:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534383.153517:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534383.153519:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534383.153520:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534383.153522:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16183 00000100:00080000:10.0:1584534383.153524:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534383.153533:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534383.153535:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534383.153535:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534383.153537:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534383.153537:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534383.153538:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534383.153539:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534383.153540:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534383.159438:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534383.159442:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534383.159444:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534383.159445:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534383.159446:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534383.159448:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534383.159449:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534383.159451:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534464 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534383.159460:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078408288 new x1661492078408320 00000800:00020000:2.0:1584534389.916164:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534389.916167:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534389.916169:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534389.916560:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534389.928615:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534383/real 1584534383] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534470 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534389.928623:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534389.928624:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534389.928625:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534389.928631:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534389.928633:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534389.928633:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534389.928636:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534389.928636:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534389.928638:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534389.928639:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534389.928640:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16189 00000100:00080000:10.0:1584534389.928642:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534389.928650:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534389.928652:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534389.928652:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534389.928653:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534389.928654:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534389.928655:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534389.928656:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534389.928657:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534389.934327:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534389.934331:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534389.934332:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534389.934333:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534389.934334:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534389.934336:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534389.934338:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534389.934339:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534470 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534389.934349:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078408320 new x1661492078408352 00000800:00000100:0.0:1584534396.141373:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534396.141786:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534396.153846:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534389/real 1584534389] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534476 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534396.153854:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534396.153855:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534396.153857:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534396.153862:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534396.153865:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534396.153873:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534396.153874:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534396.153875:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534396.153877:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534396.153877:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534396.153879:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16195 00000100:00080000:10.0:1584534396.153881:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534396.153887:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534396.153888:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534396.153888:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534396.153890:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534396.153890:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534396.153891:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534396.153892:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534396.153893:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534396.159862:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534396.159867:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534396.159868:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534396.159869:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534396.159870:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534396.159872:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534396.159874:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534396.159876:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534476 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584534396.159885:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078408352 new x1661492078408416 00000800:00000100:0.0:1584534402.141571:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534402.141986:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584534402.154037:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534396/real 1584534396] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534483 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584534402.154044:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584534402.154046:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584534402.154047:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584534402.154062:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534402.154064:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534402.154066:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534402.154067:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534402.154067:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534402.154069:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584534402.154069:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584534402.154071:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16202 00000100:00080000:12.0:1584534402.154073:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584534402.154078:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534402.154080:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534402.154080:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534402.154082:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534402.154082:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534402.154083:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534402.154084:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534402.154085:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534402.208679:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534402.208683:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534402.208685:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534402.208685:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534402.208687:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534402.208689:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534402.208690:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534402.208692:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534483 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584534402.208700:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078408416 new x1661492078408464 00000800:00020000:4.0:1584534408.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584534408.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584534408.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534408.916535:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584534408.928590:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534402/real 1584534402] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534489 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584534408.928598:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584534408.928599:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584534408.928600:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584534408.928607:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534408.928609:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534408.928609:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534408.928611:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534408.928611:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534408.928613:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584534408.928613:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584534408.928615:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16208 00000100:00080000:12.0:1584534408.928617:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584534408.928626:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534408.928628:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534408.928628:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534408.928629:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534408.928630:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534408.928631:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534408.928632:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534408.928633:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534408.981290:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534408.981295:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534408.981296:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534408.981297:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534408.981299:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534408.981300:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534408.981302:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534408.981304:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534489 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584534408.981313:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078408464 new x1661492078408512 00000800:00000100:0.0:1584534415.141998:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534415.142411:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584534415.154462:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534408/real 1584534408] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534495 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584534415.154469:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584534415.154472:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584534415.154473:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584534415.154478:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534415.154480:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534415.154481:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534415.154482:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534415.154483:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534415.154484:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584534415.154485:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584534415.154486:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16214 00000100:00080000:12.0:1584534415.154488:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584534415.154496:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534415.154497:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534415.154498:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534415.154499:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534415.154500:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534415.154501:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534415.154501:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534415.154503:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534415.201775:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534415.201779:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534415.201781:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534415.201782:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534415.201783:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534415.201785:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534415.201786:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534415.201788:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534495 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584534415.201796:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078408512 new x1661492078408544 00000800:00000100:0.0:1584534421.142286:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534421.142715:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584534421.154779:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534415/real 1584534415] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534502 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584534421.154788:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584534421.154789:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584534421.154790:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584534421.154796:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534421.154797:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534421.154806:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534421.154808:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534421.154808:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534421.154810:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584534421.154810:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584534421.154812:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16221 00000100:00080000:12.0:1584534421.154814:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584534421.154819:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534421.154821:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534421.154821:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534421.154822:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534421.154823:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534421.154824:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534421.154825:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534421.154826:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534421.186568:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534421.186573:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534421.186574:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534421.186575:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534421.186576:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534421.186578:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534421.186580:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534421.186581:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534502 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584534421.186611:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078408544 new x1661492078408592 00000800:00020000:2.0:1584534427.916083:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534427.916087:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534427.916089:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534427.916558:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584534427.928701:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534421/real 1584534421] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534508 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584534427.928713:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584534427.928714:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584534427.928717:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584534427.928740:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534427.928742:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534427.928743:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534427.928744:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534427.928745:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534427.928747:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584534427.928748:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584534427.928751:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16227 00000100:00080000:12.0:1584534427.928754:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584534427.928762:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534427.928764:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534427.928764:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534427.928766:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534427.928766:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534427.928767:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534427.928769:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534427.928771:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534427.986440:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534427.986448:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534427.986450:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534427.986452:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534427.986454:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534427.986457:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534427.986458:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534427.986460:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534508 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584534427.986491:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078408592 new x1661492078408656 00000800:00000100:0.0:1584534434.140941:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534434.141463:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584534434.153522:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534427/real 1584534427] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534514 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584534434.153531:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584534434.153533:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584534434.153535:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584534434.153542:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534434.153544:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534434.153545:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534434.153547:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534434.153547:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534434.153549:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584534434.153550:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584534434.153552:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16233 00000100:00080000:12.0:1584534434.153554:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584534434.153570:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534434.153571:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534434.153572:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534434.153573:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534434.153575:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534434.153576:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534434.153577:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534434.153578:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534434.208643:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534434.208650:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534434.208652:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534434.208653:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534434.208655:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534434.208657:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534434.208659:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534434.208661:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534514 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584534434.208670:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078408656 new x1661492078408688 00000800:00000100:0.0:1584534440.142230:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534440.142651:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584534440.154706:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534434/real 1584534434] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534521 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584534440.154715:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584534440.154716:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584534440.154717:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584534440.154723:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534440.154725:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534440.154726:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534440.154727:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534440.154727:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534440.154729:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584534440.154729:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584534440.154731:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16240 00000100:00080000:12.0:1584534440.154733:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584534440.154743:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534440.154744:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534440.154746:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534440.154747:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534440.154747:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534440.154749:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534440.154749:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534440.154750:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534440.215165:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534440.215170:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534440.215172:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534440.215173:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534440.215175:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534440.215177:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534440.215179:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534440.215180:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534521 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584534440.215188:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078408688 new x1661492078408736 00000800:00020000:6.0:1584534446.916104:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584534446.916110:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (3): c: 6, oc: 0, rc: 8 00000800:00000100:6.0:1584534446.916114:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534446.916542:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584534446.928626:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534440/real 1584534440] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534527 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584534446.928644:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584534446.928645:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584534446.928646:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584534446.928652:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534446.928654:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534446.928654:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534446.928655:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534446.928656:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534446.928657:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584534446.928658:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584534446.928659:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16246 00000100:00080000:12.0:1584534446.928661:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584534446.928670:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534446.928672:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534446.928672:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534446.928674:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534446.928674:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534446.928675:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534446.928676:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534446.928677:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534446.982070:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534446.982074:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534446.982076:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534446.982076:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534446.982079:0:25724:0:(client.c:2816:ptlrpc_resend_req()) @@@ going to resend req@ffff8d2babe45100 x1661492078408752/t0(0) o17->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 456/432 e 0 to 0 dl 1584534519 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000100:00080000:22.0:1584534446.982084:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534446.982086:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534446.982088:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534446.982090:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534527 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584534446.982119:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078408736 new x1661492078408800 00000800:00000100:0.0:1584534453.141764:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534453.142283:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584534453.154351:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534446/real 1584534446] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534533 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584534453.154359:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584534453.154360:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584534453.154361:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584534453.154374:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534453.154376:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534453.154377:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534453.154378:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534453.154379:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534453.154380:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584534453.154381:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584534453.154383:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16252 00000100:00080000:12.0:1584534453.154385:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584534453.154390:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534453.154391:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534453.154392:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534453.154393:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534453.154394:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534453.154395:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534453.154395:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534453.154397:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534453.209436:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534453.209440:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534453.209442:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534453.209442:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534453.209444:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534453.209446:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534453.209447:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534453.209449:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534533 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584534453.209478:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078408800 new x1661492078408864 00000800:00000100:0.0:1584534459.141878:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534459.142310:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584534459.154377:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534453/real 1584534453] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534540 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584534459.154385:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584534459.154386:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584534459.154387:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584534459.154393:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534459.154394:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534459.154395:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534459.154396:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534459.154397:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534459.154399:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584534459.154400:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584534459.154401:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16259 00000100:00080000:12.0:1584534459.154403:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584534459.154412:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534459.154413:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534459.154414:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534459.154415:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534459.154415:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534459.154416:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534459.154417:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534459.154418:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534459.193006:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534459.193009:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534459.193011:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534459.193012:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534459.193013:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534459.193015:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534459.193016:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534459.193018:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534540 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584534459.193048:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078408864 new x1661492078408896 00000800:00020000:2.0:1584534465.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534465.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534465.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534465.916488:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584534465.928541:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534459/real 1584534459] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534546 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584534465.928548:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584534465.928550:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584534465.928551:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584534465.928557:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534465.928559:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534465.928567:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534465.928569:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534465.928570:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534465.928571:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584534465.928572:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584534465.928574:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16265 00000100:00080000:12.0:1584534465.928575:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584534465.928581:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534465.928582:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534465.928583:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534465.928584:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534465.928584:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534465.928586:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534465.928586:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534465.928587:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534465.975671:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534465.975675:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534465.975676:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534465.975677:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534465.975678:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534465.975680:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534465.975682:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534465.975684:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534546 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584534465.975713:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078408896 new x1661492078408944 00000800:00000100:0.0:1584534472.142359:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534472.142789:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584534472.154843:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534465/real 1584534465] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534552 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584534472.154851:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584534472.154852:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584534472.154853:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584534472.154859:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534472.154862:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534472.154863:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534472.154864:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534472.154864:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534472.154866:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584534472.154867:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584534472.154868:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16271 00000100:00080000:12.0:1584534472.154870:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584534472.154878:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534472.154880:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534472.154880:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534472.154882:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534472.154882:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534472.154883:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534472.154884:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534472.154885:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534472.208939:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534472.208944:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534472.208945:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534472.208946:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534472.208947:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534472.208949:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534472.208950:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534472.208952:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534552 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584534472.208981:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078408944 new x1661492078408992 00000800:00000100:0.0:1584534478.142636:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534478.143065:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584534478.155115:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534472/real 1584534472] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534559 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584534478.155122:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584534478.155123:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584534478.155125:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584534478.155138:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534478.155140:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534478.155140:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534478.155142:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534478.155142:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534478.155144:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584534478.155144:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584534478.155146:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16278 00000100:00080000:12.0:1584534478.155148:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584534478.155152:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534478.155154:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534478.155154:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534478.155155:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534478.155156:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534478.155157:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534478.155158:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534478.155159:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534478.213518:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534478.213522:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534478.213523:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534478.213524:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534478.213525:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534478.213527:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534478.213529:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534478.213530:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534559 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584534478.213559:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078408992 new x1661492078409040 00000800:00000100:0.0:1584534484.143729:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534484.144227:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584534484.156280:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534478/real 1584534478] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534565 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584534484.156294:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584534484.156296:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584534484.156298:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584534484.156306:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534484.156308:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534484.156309:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534484.156310:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534484.156311:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534484.156312:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584534484.156313:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584534484.156315:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16284 00000100:00080000:12.0:1584534484.156319:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584534484.156334:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534484.156335:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534484.156336:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534484.156337:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534484.156338:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534484.156339:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534484.156339:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534484.156352:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534484.189140:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534484.189147:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534484.189149:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534484.189150:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534484.189152:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534484.189155:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534484.189157:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534484.189159:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534565 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584534484.189188:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078409040 new x1661492078409088 00000800:00020000:2.0:1584534490.916057:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534490.916064:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534490.916069:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534490.916541:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584534490.928624:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534484/real 1584534484] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534571 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584534490.928646:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584534490.928648:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584534490.928650:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584534490.928658:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534490.928660:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534490.928675:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534490.928676:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534490.928677:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534490.928679:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584534490.928680:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584534490.928693:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16290 00000100:00080000:12.0:1584534490.928697:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584534490.928704:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534490.928705:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534490.928706:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534490.928708:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534490.928708:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534490.928709:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534490.928711:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534490.928713:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534490.974270:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534490.974277:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534490.974280:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534490.974281:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534490.974283:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534490.974285:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534490.974287:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534490.974289:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534571 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584534490.974318:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078409088 new x1661492078409136 00000800:00000100:0.0:1584534497.140808:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534497.141278:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584534497.153354:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534490/real 1584534490] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534577 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584534497.153362:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584534497.153363:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584534497.153364:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584534497.153370:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534497.153372:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534497.153373:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534497.153374:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534497.153375:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534497.153376:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584534497.153376:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584534497.153378:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16296 00000100:00080000:12.0:1584534497.153380:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584534497.153389:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534497.153390:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534497.153391:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534497.153392:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534497.153393:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534497.153394:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534497.153406:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534497.153408:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534497.194470:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534497.194475:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534497.194477:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534497.194478:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534497.194480:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534497.194482:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534497.194484:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534497.194485:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534577 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584534497.194515:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078409136 new x1661492078409184 00000800:00000100:0.0:1584534503.140968:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534503.141413:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584534503.153484:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534497/real 1584534497] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534584 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584534503.153491:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584534503.153492:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584534503.153493:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584534503.153505:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534503.153507:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534503.153507:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534503.153509:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534503.153509:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534503.153510:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584534503.153522:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584534503.153524:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16303 00000100:00080000:12.0:1584534503.153526:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584534503.153531:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534503.153543:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534503.153544:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534503.153546:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534503.153546:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534503.153547:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534503.153548:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534503.153549:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534503.185057:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534503.185061:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534503.185062:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534503.185063:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534503.185065:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534503.185067:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534503.185069:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534503.185071:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534584 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584534503.185105:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078409184 new x1661492078409232 00000800:00020000:2.0:1584534509.916119:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534509.916122:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534509.916124:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534509.916537:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584534509.928591:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534503/real 1584534503] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534590 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584534509.928599:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584534509.928600:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584534509.928602:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584534509.928608:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534509.928609:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534509.928610:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534509.928612:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534509.928612:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534509.928614:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584534509.928614:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584534509.928616:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16309 00000100:00080000:12.0:1584534509.928618:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584534509.928627:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534509.928628:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534509.928629:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534509.928630:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534509.928631:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534509.928632:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534509.928633:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534509.928634:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534509.977718:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534509.977722:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534509.977723:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534509.977724:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534509.977726:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534509.977728:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534509.977729:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534509.977731:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534590 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584534509.977760:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078409232 new x1661492078409264 00000800:00000100:0.0:1584534516.141473:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534516.141963:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584534516.154014:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534509/real 1584534509] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534596 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584534516.154022:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584534516.154023:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584534516.154024:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584534516.154055:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534516.154057:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534516.154065:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534516.154067:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534516.154067:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534516.154069:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584534516.154069:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584534516.154072:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16315 00000100:00080000:12.0:1584534516.154074:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584534516.154079:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534516.154080:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534516.154081:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534516.154082:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534516.154083:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534516.154084:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534516.154084:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534516.154086:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534516.195904:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534516.195908:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534516.195910:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534516.195911:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534516.195912:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534516.195914:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534516.195916:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534516.195917:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534596 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584534516.195926:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078409264 new x1661492078409328 00000100:00000400:18.0:1584534519.154131:0:25734:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584534443/real 1584534443] req@ffff8d2babe40d80 x1661492078408768/t0(0) o17->public1-OST0001-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 456/432 e 0 to 1 dl 1584534519 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 00000100:02000400:18.0:1584534519.154139:0:25734:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection to public1-OST0001 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584534519.154140:0:25734:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584534519.154141:0:25734:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0001_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584534519.154147:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584534519.154149:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584534519.154149:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584534519.154151:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:18.0:1584534519.154151:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584534519.154153:0:4066:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 15152 00000100:00080000:18.0:1584534519.154154:0:4066:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584534519.154163:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584534519.154165:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584534519.154166:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584534519.154167:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584534519.154168:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584534519.154169:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584534519.154169:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584534519.154170:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584534519.154171:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584534519.154172:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000800:00000100:0.0:1584534522.141660:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534522.142070:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584534522.154120:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534516/real 1584534516] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534603 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584534522.154128:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584534522.154129:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584534522.154130:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584534522.154136:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534522.154138:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534522.154139:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534522.154140:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534522.154142:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534522.154143:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584534522.154144:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584534522.154145:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16322 00000100:00080000:14.0:1584534522.154147:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584534522.154156:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534522.154158:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534522.154158:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534522.154160:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534522.154160:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534522.154161:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534522.154162:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534522.154163:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534522.196837:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534522.196841:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534522.196842:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534522.196843:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534522.196844:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534522.196846:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534522.196849:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534522.196851:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534603 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584534522.196880:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078409328 new x1661492078409376 00000100:00080000:22.0:1584534525.154088:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:22.0:1584534525.154090:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0001_UUID on 10.10.2.22@o2ib failed (-110) 00000800:00020000:4.0:1584534528.916066:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584534528.916069:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584534528.916071:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534528.916461:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584534528.928520:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534522/real 1584534522] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534609 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584534528.928528:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584534528.928529:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584534528.928530:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584534528.928545:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534528.928547:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534528.928548:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534528.928550:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534528.928550:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534528.928552:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584534528.928552:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584534528.928554:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16328 00000100:00080000:14.0:1584534528.928556:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584534528.928561:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534528.928562:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534528.928563:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534528.928564:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534528.928565:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534528.928566:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534528.928567:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534528.928568:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534528.979742:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534528.979747:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534528.979749:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534528.979750:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534528.979751:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534528.979753:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534528.979755:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534528.979757:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534609 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584534528.979787:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078409376 new x1661492078409440 00000800:00000100:0.0:1584534535.142201:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534535.142631:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584534535.154681:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534528/real 1584534528] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534615 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584534535.154689:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584534535.154690:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584534535.154692:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584534535.154698:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534535.154700:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534535.154701:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534535.154702:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534535.154703:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534535.154704:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584534535.154705:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534535.154706:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16334 00000100:00080000:16.0:1584534535.154708:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534535.154718:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534535.154720:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534535.154721:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534535.154722:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534535.154723:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534535.154724:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534535.154724:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534535.154725:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534535.206425:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534535.206429:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534535.206430:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534535.206431:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534535.206433:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534535.206434:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534535.206436:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534535.206437:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534615 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584534535.206471:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078409440 new x1661492078409472 00000800:00000100:0.0:1584534541.142438:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534541.142896:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584534541.154950:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534535/real 1584534535] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534622 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584534541.154958:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584534541.154959:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584534541.154960:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584534541.154966:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534541.154967:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534541.154968:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:16.0:1584534541.154969:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:16.0:1584534541.154970:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534541.154972:0:4060:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16325 00000100:00080000:16.0:1584534541.154973:0:4060:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 15128 00000100:00080000:16.0:1584534541.154976:0:4060:0:(import.c:596:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0001 (at 10.10.2.21@o2ib) 00000100:00080000:16.0:1584534541.154978:0:4060:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.21@o2ib/10.10.2.21@o2ib 00000100:00080000:16.0:1584534541.154987:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534541.154988:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534541.154989:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534541.154990:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584534541.154991:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534541.154992:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16341 00000100:00080000:16.0:1584534541.154993:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534541.154996:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534541.154998:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534541.154998:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534541.154999:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534541.155000:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534541.155001:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534541.155002:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534541.155003:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534541.155231:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:22.0:1584534541.155233:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0001_UUID on 10.10.2.21@o2ib failed (-19) 00000100:00080000:22.0:1584534541.193568:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534541.193573:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534541.193575:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534541.193576:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534541.193577:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534541.193579:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534541.193580:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534541.193582:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534622 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584534541.193611:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078409472 new x1661492078409520 00000800:00020000:2.0:1584534547.916061:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534547.916064:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534547.916066:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534547.916492:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584534547.928604:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534541/real 1584534541] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534628 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584534547.928614:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584534547.928615:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584534547.928618:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584534547.928625:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534547.928627:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534547.928628:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534547.928630:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534547.928630:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534547.928632:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584534547.928633:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534547.928635:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16347 00000100:00080000:16.0:1584534547.928638:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534547.928650:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534547.928651:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534547.928652:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534547.928653:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534547.928654:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534547.928655:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534547.928656:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534547.928657:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534547.989481:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534547.989488:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534547.989490:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534547.989491:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534547.989493:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534547.989496:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534547.989497:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534547.989499:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534628 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584534547.989529:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078409520 new x1661492078409568 00000800:00000100:0.0:1584534554.140843:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534554.141250:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584534554.153307:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534547/real 1584534547] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534634 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584534554.153322:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584534554.153324:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584534554.153325:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584534554.153347:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534554.153350:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534554.153351:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534554.153352:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534554.153353:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534554.153354:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584534554.153356:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534554.153358:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16353 00000100:00080000:16.0:1584534554.153361:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534554.153368:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534554.153369:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534554.153370:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534554.153371:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534554.153372:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534554.153374:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534554.153374:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534554.153376:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534554.206629:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534554.206636:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534554.206638:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534554.206639:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534554.206641:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534554.206644:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534554.206645:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534554.206647:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534634 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584534554.206677:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078409568 new x1661492078409616 00000800:00000100:0.0:1584534560.141055:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534560.141490:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584534560.153562:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534554/real 1584534554] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534641 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584534560.153571:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584534560.153572:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584534560.153573:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584534560.153580:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534560.153581:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534560.153582:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534560.153583:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534560.153584:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534560.153585:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584534560.153586:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534560.153588:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16360 00000100:00080000:16.0:1584534560.153590:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534560.153599:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534560.153601:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534560.153601:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534560.153603:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534560.153603:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534560.153604:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534560.153606:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534560.153607:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534560.192663:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534560.192669:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534560.192671:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534560.192671:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534560.192673:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534560.192675:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534560.192677:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534560.192679:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534641 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584534560.192709:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078409616 new x1661492078409664 00000800:00020000:2.0:1584534566.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534566.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534566.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534566.916489:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584534566.928542:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534560/real 1584534560] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534647 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584534566.928550:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584534566.928551:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584534566.928552:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584534566.928556:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534566.928558:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534566.928559:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:16.0:1584534566.928561:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:16.0:1584534566.928561:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534566.928563:0:4060:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16325 00000100:00080000:16.0:1584534566.928565:0:4060:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 16347 00000100:00080000:16.0:1584534566.928566:0:4060:0:(import.c:570:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: tried all connections, increasing latency to 6s 00000100:00080000:16.0:1584534566.928568:0:4060:0:(import.c:596:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0001 (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584534566.928569:0:4060:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534566.928578:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534566.928580:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534566.928580:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534566.928581:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584534566.928583:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534566.928584:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16366 00000100:00080000:16.0:1584534566.928585:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534566.928589:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534566.928590:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534566.928591:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534566.928592:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534566.928592:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534566.928593:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534566.928594:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534566.928595:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534566.965909:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534566.965913:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534566.965915:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534566.965916:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534566.965917:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534566.965919:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534566.965920:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534566.965922:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534647 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584534566.965930:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000100:00080000:16.0:1584534566.965931:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078409664 new x1661492078409712 00000080:00080000:22.0:1584534566.965932:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534566.965933:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0001_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534566.965933:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534566.965934:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534566.965935:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534566.965936:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534566.965938:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727b00): req@ffff8d2babe40d80 x1661492078408768/t0(0) o17->public1-OST0001-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 456/432 e 0 to 1 dl 1584534519 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 00000800:00000100:0.0:1584534573.141635:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534573.142161:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584534573.154211:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534566/real 1584534566] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534653 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584534573.154219:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584534573.154220:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584534573.154221:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584534573.154227:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534573.154228:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534573.154229:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534573.154230:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534573.154232:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534573.154233:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584534573.154234:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534573.154235:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16372 00000100:00080000:16.0:1584534573.154237:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534573.154246:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534573.154247:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534573.154248:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534573.154249:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534573.154250:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534573.154251:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534573.154251:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534573.154252:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534573.203388:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534573.203393:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534573.203394:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534573.203395:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534573.203397:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534573.203398:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534573.203400:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534573.203402:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534653 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584534573.203431:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078409712 new x1661492078409760 00000800:00000100:0.0:1584534579.141986:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534579.142447:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584534579.154496:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534573/real 1584534573] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534660 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584534579.154504:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584534579.154505:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584534579.154507:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584534579.154520:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534579.154521:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534579.154523:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534579.154524:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534579.154525:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534579.154526:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584534579.154527:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534579.154529:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16379 00000100:00080000:16.0:1584534579.154531:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534579.154536:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534579.154537:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534579.154538:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534579.154539:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534579.154539:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534579.154541:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534579.154541:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534579.154542:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534579.207967:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534579.207971:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534579.207973:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534579.207973:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534579.207975:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534579.207977:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534579.207978:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534579.207980:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534660 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584534579.208009:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078409760 new x1661492078409808 00000800:00000100:0.0:1584534585.142036:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534585.142470:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584534585.154519:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534579/real 1584534579] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534666 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584534585.154527:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584534585.154528:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584534585.154529:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584534585.154537:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534585.154538:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534585.154539:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534585.154540:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534585.154541:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534585.154542:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584534585.154543:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534585.154545:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16385 00000100:00080000:16.0:1584534585.154546:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534585.154555:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534585.154556:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534585.154557:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534585.154558:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534585.154559:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534585.154560:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534585.154561:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534585.154562:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534585.211861:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534585.211865:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534585.211867:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534585.211868:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534585.211869:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534585.211871:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534585.211872:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534585.211874:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534666 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584534585.211903:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078409808 new x1661492078409840 00000800:00020000:2.0:1584534591.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534591.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534591.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534591.916571:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584534591.928624:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534585/real 1584534585] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534672 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584534591.928633:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584534591.928634:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584534591.928636:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584534591.928642:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534591.928644:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534591.928653:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534591.928655:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534591.928655:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534591.928657:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584534591.928657:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534591.928659:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16391 00000100:00080000:16.0:1584534591.928661:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534591.928667:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534591.928668:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534591.928669:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534591.928670:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534591.928671:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534591.928672:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534591.928673:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534591.928674:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534591.985138:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534591.985143:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534591.985145:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534591.985146:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534591.985147:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534591.985149:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534591.985151:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534591.985152:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534672 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584534591.985161:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078409840 new x1661492078409904 00000800:00000100:0.0:1584534598.143597:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534598.143998:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584534598.156048:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534591/real 1584534591] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534678 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584534598.156057:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584534598.156058:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584534598.156059:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584534598.156064:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534598.156066:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534598.156067:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534598.156068:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534598.156069:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534598.156070:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584534598.156071:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534598.156072:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16397 00000100:00080000:16.0:1584534598.156074:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534598.156083:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534598.156084:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534598.156085:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534598.156086:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534598.156087:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534598.156088:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534598.156088:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534598.156090:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534598.216196:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534598.216200:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534598.216202:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534598.216203:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534598.216204:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534598.216206:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534598.216207:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534598.216209:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534678 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584534598.216239:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078409904 new x1661492078409936 00000800:00000100:0.0:1584534604.143012:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534604.143468:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584534604.155512:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534598/real 1584534598] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534685 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584534604.155521:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584534604.155522:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584534604.155523:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584534604.155538:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534604.155539:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534604.155540:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534604.155541:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534604.155542:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534604.155543:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584534604.155544:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534604.155546:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16404 00000100:00080000:16.0:1584534604.155548:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534604.155553:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534604.155554:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534604.155555:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534604.155556:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534604.155557:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534604.155558:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534604.155560:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534604.155561:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534604.192382:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534604.192386:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534604.192388:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534604.192389:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534604.192390:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534604.192392:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534604.192394:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534604.192395:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534685 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584534604.192404:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078409936 new x1661492078410000 00000800:00020000:4.0:1584534610.916063:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584534610.916067:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584534610.916069:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534610.916486:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584534610.928546:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534604/real 1584534604] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534691 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584534610.928557:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584534610.928559:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584534610.928561:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584534610.928570:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534610.928572:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534610.928573:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534610.928574:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534610.928575:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534610.928577:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584534610.928578:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534610.928580:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16410 00000100:00080000:16.0:1584534610.928583:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534610.928600:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534610.928601:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534610.928602:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534610.928603:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534610.928605:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534610.928606:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534610.928607:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534610.928608:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534610.953606:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534610.953614:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534610.953616:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534610.953617:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534610.953619:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534610.953622:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534610.953623:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534610.953625:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534691 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584534610.953655:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078410000 new x1661492078410032 00000002:00080000:4.0:1584534611.648256:0:157257:0:(mdc_request.c:903:mdc_close()) @@@ matched open req@ffff8d2babe42d00 x1661492078410112/t87752872150(87752872150) o101->public1-MDT0000-mdc-ffff8d3ae8ab1800@10.10.2.11@o2ib:12/10 lens 576/616 e 0 to 0 dl 1584534618 ref 1 fl Complete:RP/4/ffffffff rc 0/-1 00000800:00000100:0.0:1584534617.141161:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534617.141606:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584534617.153663:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534610/real 1584534610] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534697 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584534617.153673:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584534617.153675:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584534617.153677:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584534617.153684:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534617.153686:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534617.153696:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534617.153697:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534617.153698:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534617.153700:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584534617.153701:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534617.153703:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16416 00000100:00080000:16.0:1584534617.153707:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534617.153715:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534617.153716:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534617.153718:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534617.153720:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534617.153721:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534617.153722:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534617.153723:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534617.153724:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534617.160101:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534617.160108:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534617.160110:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534617.160111:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534617.160113:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534617.160116:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534617.160117:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534617.160119:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534697 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584534617.160149:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078410032 new x1661492078410208 00000800:00000100:0.0:1584534623.140510:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534623.140958:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584534623.153011:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534617/real 1584534617] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534704 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584534623.153020:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584534623.153021:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584534623.153023:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584534623.153056:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534623.153058:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534623.153058:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534623.153060:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534623.153060:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534623.153062:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584534623.153063:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534623.153065:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16423 00000100:00080000:16.0:1584534623.153068:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534623.153080:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534623.153081:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534623.153082:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534623.153083:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534623.153084:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534623.153085:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534623.153086:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534623.153087:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534623.159768:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534623.159774:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534623.159776:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534623.159777:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534623.159779:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534623.159781:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534623.159783:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534623.159785:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534704 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584534623.159793:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078410208 new x1661492078410240 00000800:00020000:2.0:1584534629.916069:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534629.916072:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534629.916074:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534629.916607:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584534629.928664:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534623/real 1584534623] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534710 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584534629.928672:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584534629.928673:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584534629.928675:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584534629.928690:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534629.928691:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534629.928692:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534629.928693:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534629.928694:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534629.928696:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584534629.928696:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534629.928699:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16429 00000100:00080000:16.0:1584534629.928701:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534629.928706:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534629.928707:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534629.928708:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534629.928709:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534629.928710:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534629.928711:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534629.928712:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534629.928713:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534629.935480:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534629.935485:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534629.935487:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534629.935488:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534629.935489:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534629.935491:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534629.935492:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534629.935494:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534710 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584534629.935503:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078410240 new x1661492078410288 00000800:00000100:0.0:1584534635.141782:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534635.142299:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584534635.154347:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534629/real 1584534629] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534716 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584534635.154356:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584534635.154357:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584534635.154358:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584534635.154364:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534635.154366:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534635.154366:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534635.154368:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534635.154368:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534635.154371:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584534635.154371:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534635.154373:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16435 00000100:00080000:16.0:1584534635.154375:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534635.154384:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534635.154385:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534635.154386:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534635.154387:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534635.154388:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534635.154389:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534635.154390:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534635.154391:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534635.160330:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534635.160334:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534635.160336:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534635.160337:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534635.160338:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534635.160340:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534635.160341:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534635.160343:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534716 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584534635.160352:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078410288 new x1661492078410336 00000800:00000100:0.0:1584534641.141021:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534641.141415:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584534641.153464:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534635/real 1584534635] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534722 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584534641.153472:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584534641.153473:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584534641.153474:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584534641.153480:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534641.153482:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534641.153490:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534641.153493:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534641.153493:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534641.153495:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584534641.153495:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534641.153497:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16441 00000100:00080000:16.0:1584534641.153499:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534641.153504:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534641.153505:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534641.153506:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534641.153507:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534641.153508:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534641.153509:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534641.153510:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534641.153511:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534641.173819:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534641.173823:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534641.173824:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534641.173825:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534641.173826:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534641.173828:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534641.173830:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534641.173831:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534722 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584534641.173840:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078410336 new x1661492078410384 00000800:00000100:0.0:1584534647.141218:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534647.141632:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584534647.153690:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534641/real 1584534641] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534728 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584534647.153698:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584534647.153699:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584534647.153700:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584534647.153706:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534647.153709:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534647.153709:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534647.153711:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534647.153711:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534647.153713:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584534647.153713:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534647.153715:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16447 00000100:00080000:16.0:1584534647.153717:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534647.153726:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534647.153727:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534647.153728:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534647.153729:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534647.153729:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534647.153731:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534647.153731:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534647.153732:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534647.160343:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534647.160348:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534647.160349:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534647.160350:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534647.160351:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534647.160353:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534647.160355:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534647.160357:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534728 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584534647.160386:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078410384 new x1661492078410432 00000800:00020000:2.0:1584534653.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534653.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534653.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534653.916503:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584534653.928566:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534647/real 1584534647] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534734 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584534653.928573:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584534653.928575:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584534653.928577:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584534653.928591:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534653.928593:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534653.928594:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534653.928595:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534653.928596:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534653.928597:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584534653.928598:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534653.928599:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16453 00000100:00080000:16.0:1584534653.928601:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534653.928607:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534653.928608:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534653.928609:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534653.928610:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534653.928611:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534653.928612:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534653.928612:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534653.928614:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534653.935036:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534653.935040:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534653.935041:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534653.935042:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534653.935043:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534653.935045:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534653.935046:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534653.935048:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534734 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584534653.935080:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078410432 new x1661492078410480 00000800:00000100:0.0:1584534659.141492:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534659.141922:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584534659.153965:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534653/real 1584534653] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534740 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584534659.153978:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584534659.153979:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584534659.153980:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584534659.153986:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534659.153987:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534659.153988:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534659.153989:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534659.153990:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534659.153991:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584534659.153992:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534659.153993:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16459 00000100:00080000:16.0:1584534659.153995:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534659.154004:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534659.154005:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534659.154006:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534659.154007:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534659.154008:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534659.154009:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534659.154009:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534659.154011:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534659.162586:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534659.162590:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534659.162591:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534659.162592:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534659.162593:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534659.162595:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534659.162597:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534659.162599:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534740 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584534659.162607:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078410480 new x1661492078410512 00000800:00000100:0.0:1584534665.141606:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534665.142012:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584534665.154070:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534659/real 1584534659] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534746 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584534665.154081:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584534665.154083:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584534665.154085:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584534665.154093:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534665.154095:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534665.154109:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534665.154111:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534665.154112:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534665.154113:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584534665.154115:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534665.154117:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16465 00000100:00080000:16.0:1584534665.154120:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534665.154128:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534665.154129:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534665.154130:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534665.154132:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534665.154132:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534665.154134:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534665.154136:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534665.154137:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534665.162960:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534665.162968:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534665.162970:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534665.162972:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534665.162973:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534665.162976:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534665.162978:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534665.162980:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534746 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584534665.162989:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078410512 new x1661492078410576 00000800:00000100:0.0:1584534671.141377:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534671.141797:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584534671.153854:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534665/real 1584534665] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534752 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584534671.153865:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584534671.153866:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584534671.153868:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584534671.153876:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534671.153878:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534671.153879:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534671.153881:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534671.153881:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534671.153883:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584534671.153884:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534671.153886:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16471 00000100:00080000:16.0:1584534671.153889:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534671.153906:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534671.153908:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534671.153908:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534671.153910:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534671.153911:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534671.153913:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534671.153914:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534671.153915:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534671.184949:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534671.184956:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534671.184958:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534671.184960:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534671.184962:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534671.184964:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534671.184966:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534671.184968:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534752 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584534671.184997:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078410576 new x1661492078410608 00000800:00020000:2.0:1584534677.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534677.916081:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534677.916083:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534677.916557:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584534677.928647:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534671/real 1584534671] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534758 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584534677.928655:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584534677.928657:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584534677.928658:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584534677.928673:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534677.928675:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534677.928675:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534677.928677:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534677.928677:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534677.928679:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584534677.928679:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584534677.928681:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16477 00000100:00080000:16.0:1584534677.928683:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584534677.928689:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534677.928691:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534677.928692:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534677.928693:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534677.928694:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534677.928695:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584534677.928696:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584534677.928697:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534677.937306:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534677.937311:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534677.937313:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534677.937314:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534677.937315:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534677.937317:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534677.937319:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534677.937320:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534758 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584534677.937353:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078410608 new x1661492078410672 00000800:00000100:0.0:1584534683.141361:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534683.141814:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584534683.153870:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534677/real 1584534677] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534764 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584534683.153878:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584534683.153879:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584534683.153881:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584534683.153887:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584534683.153888:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584534683.153889:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584534683.153891:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584534683.153891:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584534683.153895:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584534683.153895:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584534683.153897:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16483 00000100:00080000:18.0:1584534683.153899:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584534683.153908:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584534683.153909:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584534683.153910:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584534683.153911:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584534683.153912:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584534683.153913:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584534683.153914:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584534683.153915:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534683.162498:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534683.162503:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534683.162504:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534683.162505:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534683.162506:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534683.162508:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534683.162510:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534683.162511:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534764 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584534683.162540:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078410672 new x1661492078410704 00000800:00000100:0.0:1584534689.141540:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534689.141975:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584534689.154025:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534683/real 1584534683] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534770 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584534689.154086:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584534689.154087:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584534689.154089:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584534689.154094:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584534689.154096:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584534689.154105:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584534689.154107:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584534689.154107:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584534689.154109:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584534689.154109:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584534689.154111:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16489 00000100:00080000:18.0:1584534689.154113:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584534689.154119:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584534689.154120:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584534689.154120:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584534689.154122:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584534689.154122:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584534689.154123:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584534689.154124:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584534689.154125:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534689.162969:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534689.162973:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534689.162974:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534689.162975:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534689.162976:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534689.162978:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534689.162979:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534689.162981:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534770 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584534689.163011:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078410704 new x1661492078410752 00000800:00000100:0.0:1584534695.141759:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534695.142271:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584534695.154339:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534689/real 1584534689] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534776 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584534695.154347:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584534695.154348:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584534695.154350:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584534695.154356:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534695.154357:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534695.154358:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534695.154359:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534695.154360:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534695.154361:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584534695.154362:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584534695.154364:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16495 00000100:00080000:20.0:1584534695.154367:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584534695.154377:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534695.154378:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534695.154379:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534695.154380:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534695.154381:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534695.154382:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534695.154382:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534695.154384:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534695.187594:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534695.187599:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534695.187600:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534695.187601:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534695.187602:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534695.187604:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534695.187606:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534695.187607:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534776 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584534695.187637:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078410752 new x1661492078410800 00000800:00020000:6.0:1584534701.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584534701.916081:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584534701.916083:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534701.916520:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584534701.928592:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534695/real 1584534695] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534782 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584534701.928600:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584534701.928601:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584534701.928602:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584534701.928615:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534701.928617:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534701.928617:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534701.928619:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534701.928619:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534701.928620:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584534701.928622:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584534701.928624:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16501 00000100:00080000:20.0:1584534701.928626:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584534701.928631:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534701.928633:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534701.928633:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534701.928634:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534701.928635:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534701.928636:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534701.928637:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534701.928638:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534701.937710:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534701.937715:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534701.937716:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534701.937717:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534701.937718:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534701.937720:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534701.937721:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534701.937723:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534782 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584534701.937757:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078410800 new x1661492078410848 00000800:00000100:0.0:1584534707.142232:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534707.142651:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584534707.154726:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534701/real 1584534701] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534788 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584534707.154733:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584534707.154734:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584534707.154735:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584534707.154741:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534707.154742:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534707.154743:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534707.154744:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534707.154745:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534707.154746:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584534707.154747:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584534707.154748:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16507 00000100:00080000:20.0:1584534707.154750:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584534707.154758:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534707.154759:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534707.154759:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534707.154761:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534707.154761:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534707.154762:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534707.154763:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534707.154775:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534707.163682:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534707.163686:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534707.163688:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534707.163689:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534707.163690:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534707.163692:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534707.163693:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534707.163695:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534788 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584534707.163728:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078410848 new x1661492078410896 00000800:00000100:0.0:1584534713.142516:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534713.142948:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584534713.154999:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534707/real 1584534707] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534794 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584534713.155006:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584534713.155008:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584534713.155009:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584534713.155015:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534713.155016:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534713.155025:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534713.155037:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534713.155037:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534713.155039:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584534713.155039:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584534713.155041:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16513 00000100:00080000:20.0:1584534713.155043:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584534713.155049:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534713.155050:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534713.155051:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534713.155052:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534713.155053:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534713.155054:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534713.155054:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534713.155055:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534713.180850:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534713.180854:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534713.180855:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534713.180856:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534713.180858:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534713.180860:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534713.180861:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534713.180863:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534794 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584534713.180892:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078410896 new x1661492078410944 00000800:00000100:0.0:1584534719.142627:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534719.143147:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584534719.155212:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534713/real 1584534713] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534800 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584534719.155219:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584534719.155220:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584534719.155221:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584534719.155227:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534719.155228:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534719.155229:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534719.155230:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534719.155230:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534719.155231:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584534719.155232:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584534719.155233:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16519 00000100:00080000:20.0:1584534719.155235:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584534719.155242:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534719.155243:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534719.155244:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534719.155245:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534719.155246:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534719.155247:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534719.155247:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534719.155248:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534719.182885:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534719.182890:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534719.182891:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534719.182892:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534719.182893:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534719.182895:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534719.182897:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534719.182898:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534800 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584534719.182932:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078410944 new x1661492078410976 00000800:00020000:2.0:1584534725.916070:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534725.916073:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534725.916075:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534725.916498:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584534725.928554:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534719/real 1584534719] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534806 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584534725.928564:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584534725.928566:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584534725.928568:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584534725.928586:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534725.928588:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534725.928589:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534725.928590:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534725.928591:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534725.928593:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584534725.928593:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584534725.928595:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16525 00000100:00080000:20.0:1584534725.928598:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584534725.928605:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534725.928607:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534725.928607:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534725.928608:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534725.928609:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534725.928610:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534725.928611:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534725.928612:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534725.955136:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534725.955143:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534725.955145:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534725.955146:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534725.955148:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534725.955150:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534725.955152:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534725.955154:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534806 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584534725.955183:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078410976 new x1661492078411040 00000800:00000100:0.0:1584534732.141660:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534732.142168:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584534732.154276:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534725/real 1584534725] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534812 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584534732.154286:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584534732.154288:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584534732.154290:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584534732.154298:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534732.154301:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534732.154302:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534732.154303:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534732.154304:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534732.154306:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584534732.154307:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584534732.154309:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16531 00000100:00080000:20.0:1584534732.154312:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584534732.154327:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534732.154329:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534732.154329:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534732.154331:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534732.154331:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534732.154332:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534732.154333:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534732.154334:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534732.182956:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534732.182963:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534732.182966:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534732.182967:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534732.182969:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534732.182971:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534732.182973:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534732.182975:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534812 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584534732.183004:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078411040 new x1661492078411072 00000800:00000100:0.0:1584534738.140738:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534738.141241:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584534738.153313:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534732/real 1584534732] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534819 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584534738.153323:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584534738.153325:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584534738.153327:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584534738.153334:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534738.153336:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534738.153347:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534738.153349:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534738.153349:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534738.153351:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584534738.153352:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584534738.153354:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16538 00000100:00080000:20.0:1584534738.153358:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584534738.153365:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534738.153367:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534738.153368:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534738.153369:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534738.153370:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534738.153371:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534738.153373:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534738.153374:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534738.185222:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534738.185229:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534738.185232:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534738.185233:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534738.185235:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534738.185237:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534738.185239:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534738.185241:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534819 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584534738.185251:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078411072 new x1661492078411136 00000800:00020000:4.0:1584534744.916067:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584534744.916070:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584534744.916073:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534744.916495:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584534744.928552:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534738/real 1584534738] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534825 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584534744.972336:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584534744.972337:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584534744.972351:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584534744.972356:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534744.972358:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534744.972359:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534744.972360:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534744.972361:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534744.972362:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584534744.972363:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584534744.972365:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16544 00000100:00080000:20.0:1584534744.972367:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584534744.972378:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534744.972380:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534744.972380:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534744.972382:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534744.972383:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534744.972385:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534744.972385:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534744.972387:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534745.000261:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534745.000267:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534745.000269:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534745.000270:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534745.000272:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534745.000274:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534745.000276:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534745.000278:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534825 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584534745.000307:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078411136 new x1661492078411168 00000800:00000100:0.0:1584534751.141313:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534751.141846:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584534751.153912:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534744/real 1584534744] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534831 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584534751.153919:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584534751.153920:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584534751.153921:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584534751.153934:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534751.153935:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534751.153936:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534751.153937:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534751.153937:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534751.153939:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584534751.153950:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584534751.153952:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16550 00000100:00080000:20.0:1584534751.153954:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584534751.153958:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534751.153971:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534751.153972:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534751.153974:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534751.153974:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534751.153975:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534751.153976:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534751.153977:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534751.192429:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534751.192433:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534751.192434:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534751.192435:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534751.192437:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534751.192438:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534751.192440:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534751.192441:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534831 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584534751.192476:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078411168 new x1661492078411216 00000800:00000100:0.0:1584534757.141498:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534757.141894:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584534757.153945:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534751/real 1584534751] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534838 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584534757.153953:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584534757.153954:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584534757.153955:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584534757.153961:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534757.153962:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534757.153963:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534757.153964:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534757.153965:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534757.153966:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584534757.153967:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584534757.153969:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16557 00000100:00080000:20.0:1584534757.153971:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584534757.153981:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534757.153983:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534757.153984:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534757.153985:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534757.153985:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534757.153986:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534757.153987:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534757.153988:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534757.190996:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534757.191001:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534757.191003:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534757.191003:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534757.191005:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534757.191007:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534757.191008:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534757.191010:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534838 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584534757.191040:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078411216 new x1661492078411264 00000800:00000100:0.0:1584534763.141761:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534763.142271:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584534763.154351:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534757/real 1584534757] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534844 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584534763.154360:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584534763.154362:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584534763.154364:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584534763.154370:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534763.154372:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534763.154387:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534763.154388:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534763.154389:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534763.154390:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584534763.154391:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584534763.154405:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16563 00000100:00080000:20.0:1584534763.154409:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584534763.154427:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534763.154428:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534763.154430:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534763.154431:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534763.154432:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534763.154433:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534763.154433:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534763.154446:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534763.185251:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534763.185258:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534763.185260:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534763.185261:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534763.185263:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534763.185266:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534763.185267:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534763.185269:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534844 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584534763.185299:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078411264 new x1661492078411312 00000800:00000100:0.0:1584534769.141970:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534769.142472:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584534769.154526:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534763/real 1584534763] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534850 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584534769.154535:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584534769.154536:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584534769.154538:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584534769.154544:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534769.154546:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534769.154547:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534769.154548:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534769.154549:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534769.154552:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584534769.154553:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584534769.154555:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16569 00000100:00080000:20.0:1584534769.154557:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584534769.154570:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534769.154571:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534769.154572:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534769.154573:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534769.154574:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534769.154575:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534769.154576:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534769.154577:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534769.182241:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534769.182247:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534769.182249:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534769.182250:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534769.182252:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534769.182255:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534769.182256:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534769.182258:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534850 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584534769.182291:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078411312 new x1661492078411360 00000800:00000100:0.0:1584534775.142262:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534775.142759:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584534775.154827:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534769/real 1584534769] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534856 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584534775.154835:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584534775.154836:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584534775.154838:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584534775.154853:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534775.154854:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534775.154855:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534775.154860:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534775.154861:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534775.154862:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584534775.154863:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584534775.154864:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16575 00000100:00080000:20.0:1584534775.154867:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584534775.154872:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534775.154874:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534775.154874:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534775.154876:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534775.154876:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534775.154877:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534775.154878:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534775.154879:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534775.189346:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534775.189351:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534775.189353:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534775.189353:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534775.189355:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534775.189357:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534775.189358:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534775.189360:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534856 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584534775.189389:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078411360 new x1661492078411408 00000800:00000100:0.0:1584534781.142508:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534781.143041:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584534781.155109:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534775/real 1584534775] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534862 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584534781.155117:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584534781.155118:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584534781.155120:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584534781.155125:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534781.155128:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534781.155129:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534781.155130:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534781.155130:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534781.155132:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584534781.155132:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584534781.155134:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16581 00000100:00080000:20.0:1584534781.155136:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584534781.155144:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534781.155146:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534781.155146:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534781.155147:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534781.155148:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534781.155149:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534781.155150:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534781.155151:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534781.194115:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534781.194119:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534781.194121:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534781.194122:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534781.194123:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534781.194125:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534781.194126:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534781.194128:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534862 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584534781.194157:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078411408 new x1661492078411440 00000800:00000100:0.0:1584534787.142712:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534787.143187:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584534787.155297:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534781/real 1584534781] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534868 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584534787.155306:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584534787.155307:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584534787.155311:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584534787.155318:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534787.155320:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534787.155332:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534787.155333:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534787.155334:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534787.155336:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584534787.155336:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584534787.155339:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16587 00000100:00080000:20.0:1584534787.155353:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584534787.155361:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534787.155362:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534787.155363:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534787.155364:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534787.155365:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534787.155366:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534787.155367:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534787.155368:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534787.200417:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534787.200423:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534787.200426:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534787.200427:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534787.200428:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534787.200431:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534787.200432:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534787.200434:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534868 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584534787.200463:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078411440 new x1661492078411504 00000800:00020000:2.0:1584534793.916054:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534793.916058:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534793.916060:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534793.916501:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584534793.928559:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534787/real 1584534787] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534874 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584534793.928574:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584534793.928576:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584534793.928578:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584534793.928586:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534793.928589:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534793.928590:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534793.928591:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534793.928592:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534793.928593:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584534793.928594:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584534793.928596:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16593 00000100:00080000:20.0:1584534793.928600:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584534793.928616:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534793.928618:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534793.928619:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534793.928620:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534793.928620:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534793.928622:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534793.928622:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534793.928624:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534793.964288:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534793.964295:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534793.964297:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534793.964298:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534793.964300:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534793.964303:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534793.964305:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534793.964306:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534874 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584534793.964340:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078411504 new x1661492078411536 00000800:00000100:0.0:1584534800.140945:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534800.141448:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584534800.153498:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534793/real 1584534793] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534880 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584534800.153506:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584534800.153507:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584534800.153509:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584534800.153522:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534800.153524:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534800.153525:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534800.153527:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534800.153527:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534800.153529:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584534800.153530:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584534800.153532:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16599 00000100:00080000:20.0:1584534800.153534:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584534800.153541:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534800.153542:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534800.153543:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534800.153544:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534800.153544:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534800.153546:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534800.153547:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534800.153549:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534800.182966:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534800.182971:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534800.182973:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534800.182974:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534800.182976:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534800.182978:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534800.182980:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534800.182981:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534880 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584534800.183011:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078411536 new x1661492078411600 00000800:00000100:0.0:1584534806.141244:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534806.141733:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584534806.153801:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534800/real 1584534800] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534887 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584534806.153809:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584534806.153810:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584534806.153812:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584534806.153818:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534806.153819:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534806.153820:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534806.153821:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534806.153822:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534806.153823:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584534806.153824:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584534806.153825:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16606 00000100:00080000:20.0:1584534806.153827:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584534806.153836:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534806.153837:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534806.153838:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534806.153839:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534806.153840:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534806.153841:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534806.153842:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534806.153843:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534806.192094:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534806.192098:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534806.192099:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534806.192100:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534806.192101:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534806.192103:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534806.192105:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534806.192106:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534887 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584534806.192143:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078411600 new x1661492078411632 00000800:00020000:4.0:1584534812.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584534812.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584534812.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534812.916588:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584534812.928642:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534806/real 1584534806] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534893 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584534812.928650:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584534812.928651:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584534812.928653:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584534812.928658:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534812.928659:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534812.928667:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534812.928669:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534812.928669:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534812.928671:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584534812.928671:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584534812.928673:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16612 00000100:00080000:20.0:1584534812.928675:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584534812.928681:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534812.928682:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534812.928683:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534812.928684:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534812.928685:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534812.928686:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534812.928687:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534812.928688:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534812.970625:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534812.970631:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534812.970632:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534812.970633:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534812.970634:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534812.970636:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534812.970638:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534812.970639:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534893 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584534812.970647:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078411632 new x1661492078411696 00000800:00000100:0.0:1584534819.141650:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534819.142081:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584534819.154131:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534812/real 1584534812] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534899 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584534819.154138:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584534819.179481:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584534819.179483:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584534819.179487:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534819.179488:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534819.179489:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534819.179490:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534819.179491:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534819.179492:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584534819.179492:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584534819.179494:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16618 00000100:00080000:20.0:1584534819.179496:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584534819.179507:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534819.179508:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534819.179509:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534819.179510:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534819.179510:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534819.179512:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584534819.179512:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584534819.179513:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534819.230644:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534819.230648:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534819.230649:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534819.230650:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534819.230651:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534819.230653:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534819.230655:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534819.248456:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534899 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:0.0:1584534819.248502:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078411696 new x1661492078411728 00000800:00000100:0.0:1584534825.141844:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534825.142293:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584534825.154370:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534819/real 1584534819] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534906 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584534825.154378:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584534825.154379:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584534825.154380:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584534825.154395:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584534825.154397:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584534825.154397:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584534825.154398:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584534825.154399:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584534825.154400:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584534825.154401:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584534825.154403:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16625 00000100:00080000:4.0:1584534825.154406:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584534825.154411:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584534825.154413:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584534825.154413:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584534825.154415:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584534825.154415:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584534825.154416:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584534825.154417:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584534825.154418:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534825.192974:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534825.192978:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534825.192979:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534825.192980:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534825.192981:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534825.192983:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534825.192985:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534825.192987:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534906 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584534825.193016:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078411728 new x1661492078411776 00000800:00020000:4.0:1584534831.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584534831.916081:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584534831.916083:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534831.916491:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584534831.928551:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534825/real 1584534825] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534912 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584534831.928559:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584534831.928560:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584534831.928562:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584534831.928569:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584534831.928571:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584534831.928571:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584534831.928573:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584534831.928574:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584534831.928576:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584534831.928577:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584534831.928579:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16631 00000100:00080000:6.0:1584534831.928581:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584534831.928592:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584534831.928595:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584534831.928596:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584534831.928598:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584534831.928599:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584534831.928601:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584534831.928602:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584534831.928603:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534831.968421:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534831.968427:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534831.968428:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534831.968429:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534831.968431:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534831.968433:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534831.968434:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534831.968436:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534912 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584534831.968465:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078411776 new x1661492078411824 00000800:00000100:0.0:1584534838.142468:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534838.142935:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584534838.154985:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534831/real 1584534831] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534918 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584534838.154993:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584534838.154994:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584534838.154995:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584534838.155000:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584534838.155001:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584534838.155011:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584534838.155012:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584534838.155012:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584534838.155014:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584534838.155015:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584534838.155016:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16637 00000100:00080000:6.0:1584534838.155018:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584534838.155024:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584534838.155025:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584534838.155026:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584534838.155058:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584534838.155058:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584534838.155059:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584534838.155060:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584534838.155061:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534838.195955:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534838.195959:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534838.195960:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534838.195961:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534838.195962:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534838.195964:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534838.195966:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534838.195967:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534918 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534838.195975:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078411824 new x1661492078411872 00000800:00000100:0.0:1584534844.142790:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534844.143256:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534844.155312:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534838/real 1584534838] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534925 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534844.155320:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534844.155322:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534844.155324:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534844.155331:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534844.155332:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534844.155333:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534844.155334:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534844.155335:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534844.155337:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534844.155337:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534844.155339:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16644 00000100:00080000:10.0:1584534844.155341:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534844.155352:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534844.155353:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534844.155354:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534844.155355:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534844.155355:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534844.155357:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534844.155357:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534844.155358:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534846.546129:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534846.546134:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534846.546136:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534846.546136:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534846.546138:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534846.546140:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534846.546142:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534846.546143:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534925 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534846.546155:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078411872 new x1661492078411920 00000800:00020000:6.0:1584534851.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584534851.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584534851.916086:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534851.916614:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534851.928679:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534846/real 1584534846] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534934 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534851.928696:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534851.928697:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534851.928699:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534851.928723:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534851.928725:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534851.928726:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534851.928728:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534851.928728:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534851.928730:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534851.928731:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534851.928734:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16650 00000100:00080000:10.0:1584534851.928737:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534851.928745:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534851.928746:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534851.928747:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534851.928749:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534851.928749:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534851.928751:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534851.928751:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534851.928753:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534851.972623:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534851.972631:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534851.972634:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534851.972635:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534851.972637:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534851.972639:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534851.972641:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534851.972643:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534934 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534851.972653:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078411920 new x1661492078411968 00000800:00020000:4.0:1584534857.916061:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584534857.916065:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584534857.916067:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534857.916512:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534857.928561:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534851/real 1584534851] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534938 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534857.928571:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534857.928573:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534857.928574:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534857.928581:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534857.928583:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534857.928584:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534857.928585:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534857.928586:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534857.928588:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534857.928589:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534857.928591:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16657 00000100:00080000:10.0:1584534857.928593:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534857.928604:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534857.928606:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534857.928606:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534857.928608:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534857.928608:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534857.928610:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534857.928611:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534857.928613:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534857.982050:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534857.982057:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534857.982059:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534857.982060:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534857.982062:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534857.982064:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534857.982066:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534857.982068:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534938 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534857.982077:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078411968 new x1661492078412016 00000800:00020000:0.0:1584534863.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584534863.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584534863.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534863.916489:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534863.928539:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534857/real 1584534857] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534944 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534863.928547:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534863.928548:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534863.928549:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534863.928555:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534863.928557:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534863.928566:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534863.928567:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534863.928568:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534863.928569:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534863.928570:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534863.928571:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16663 00000100:00080000:10.0:1584534863.928573:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534863.928578:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534863.928580:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534863.928580:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534863.928581:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534863.928583:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534863.928584:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534863.928585:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534863.928586:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534863.977042:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534863.977046:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534863.977048:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534863.977048:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534863.977050:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534863.977052:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534863.977053:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534863.977055:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534944 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534863.977063:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078412016 new x1661492078412064 00000800:00020000:0.0:1584534869.916057:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584534869.916060:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584534869.916062:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534869.916575:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534869.928629:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534863/real 1584534863] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534950 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534869.928636:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534869.928638:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534869.928639:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534869.928645:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534869.928647:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534869.928648:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534869.928649:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534869.928650:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534869.928651:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534869.928652:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534869.928653:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16669 00000100:00080000:10.0:1584534869.928655:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534869.928665:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534869.928666:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534869.928668:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534869.928669:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534869.928670:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534869.928671:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534869.928672:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534869.928673:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584534869.989877:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584534869.989883:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584534869.989884:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584534869.989885:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584534869.989886:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584534869.989888:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584534869.989890:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584534869.989892:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534950 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534869.989921:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078412064 new x1661492078412096 00000800:00020000:0.0:1584534875.916062:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584534875.916065:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584534875.916067:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534875.916473:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534875.928528:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534869/real 1584534869] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534956 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534875.928536:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534875.928537:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534875.928539:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534875.928553:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534875.928555:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534875.928555:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534875.928557:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534875.928557:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534875.928559:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534875.928560:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534875.928561:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16675 00000100:00080000:10.0:1584534875.928564:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534875.928570:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534875.928572:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534875.928573:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534875.928574:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534875.928575:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534875.928576:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534875.928576:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534875.928577:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584534875.957472:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584534875.957476:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584534875.957477:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584534875.957478:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584534875.957479:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584534875.957481:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584534875.957483:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584534875.957485:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534956 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534875.957518:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078412096 new x1661492078412160 00000800:00020000:2.0:1584534881.916193:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534881.916196:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534881.916197:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534881.916614:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534881.928668:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534875/real 1584534875] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534962 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534881.928676:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534881.928677:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534881.928678:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534881.928684:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534881.928685:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534881.928686:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534881.928687:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534881.928688:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534881.928689:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534881.928690:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534881.928692:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16681 00000100:00080000:10.0:1584534881.928694:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534881.928702:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534881.928704:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534881.928704:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534881.928706:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534881.928706:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534881.928707:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534881.928708:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534881.928709:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584534881.944503:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584534881.944507:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584534881.944509:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584534881.944510:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584534881.944511:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584534881.944513:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584534881.944514:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584534881.944516:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534962 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534881.944544:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078412160 new x1661492078412192 00000800:00020000:4.0:1584534887.916156:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584534887.916158:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584534887.916160:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534887.916635:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534887.928693:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534881/real 1584534881] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534968 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534887.928701:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534887.928702:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534887.928703:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534887.928709:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534887.928711:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534887.928720:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534887.928721:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534887.928722:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534887.928724:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534887.928724:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534887.928726:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16687 00000100:00080000:10.0:1584534887.928728:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534887.928734:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534887.928735:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534887.928736:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534887.928737:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534887.928737:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534887.928738:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534887.928739:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534887.928740:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584534887.977227:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584534887.977231:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584534887.977233:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584534887.977233:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584534887.977235:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584534887.977236:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584534887.977238:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584534887.977240:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534968 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534887.977273:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078412192 new x1661492078412256 00000800:00020000:4.0:1584534893.916199:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584534893.916201:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584534893.916203:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534893.916670:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534893.928715:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534887/real 1584534887] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534974 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534893.928723:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534893.928724:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534893.928726:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534893.928732:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534893.928733:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534893.928734:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534893.928736:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534893.928737:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534893.928738:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534893.928739:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534893.928740:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16693 00000100:00080000:10.0:1584534893.928743:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534893.928751:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534893.928752:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534893.928753:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534893.928754:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534893.928755:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534893.928756:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534893.928756:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534893.928757:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584534893.963261:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584534893.963265:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584534893.963267:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584534893.963267:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584534893.963269:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584534893.963271:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584534893.963272:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584534893.963275:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534974 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534893.963304:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078412256 new x1661492078412288 00000800:00020000:4.0:1584534899.916200:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584534899.916203:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584534899.916204:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534899.916610:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534899.928688:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534893/real 1584534893] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534980 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534899.928696:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534899.928697:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534899.928698:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534899.928711:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534899.928714:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534899.928715:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534899.928716:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534899.928716:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534899.928718:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534899.928718:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534899.928720:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16699 00000100:00080000:10.0:1584534899.928722:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534899.928727:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534899.928728:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534899.928729:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534899.928730:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534899.928730:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534899.928732:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534899.928732:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534899.928733:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584534899.972603:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584534899.972607:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584534899.972608:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584534899.972609:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584534899.972611:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584534899.972612:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584534899.972614:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584534899.972616:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534980 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534899.972644:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078412288 new x1661492078412336 00000800:00020000:2.0:1584534905.916081:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534905.916085:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534905.916087:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534905.916542:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534905.928598:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534899/real 1584534899] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534986 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534905.928608:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534905.928610:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534905.928612:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534905.928620:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534905.928622:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534905.928623:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534905.928624:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534905.928625:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534905.928626:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534905.928627:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534905.928629:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16705 00000100:00080000:10.0:1584534905.928632:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534905.928647:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534905.928648:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534905.928649:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534905.928650:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534905.928651:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534905.928652:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534905.928652:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534905.928654:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584534905.984936:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584534905.984942:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584534905.984944:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584534905.984945:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584534905.984947:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584534905.984949:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584534905.984951:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584534905.984953:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534986 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534905.984981:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078412336 new x1661492078412384 00000800:00020000:2.0:1584534911.916055:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534911.916058:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534911.916060:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534911.916476:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534911.928566:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534905/real 1584534905] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534992 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534911.928591:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534911.928593:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534911.928595:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534911.928602:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534911.928604:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534911.928617:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534911.928619:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534911.928620:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534911.928621:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534911.928623:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534911.928625:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16711 00000100:00080000:10.0:1584534911.928627:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534911.928634:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534911.928636:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534911.928637:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534911.928638:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534911.928639:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534911.928640:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534911.928641:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534911.928642:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584534911.989652:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584534911.989659:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584534911.989662:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584534911.989663:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584534911.989665:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584534911.989667:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584534911.989668:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584534911.989671:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534992 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534911.989701:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078412384 new x1661492078412432 00000800:00020000:4.0:1584534917.916181:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584534917.916185:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584534917.916188:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534917.916638:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534917.928714:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534911/real 1584534911] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534998 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534917.928724:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534917.928725:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534917.928727:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534917.928734:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534917.928736:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534917.928736:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534917.928738:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534917.928738:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534917.928740:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534917.928741:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534917.928743:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16717 00000100:00080000:10.0:1584534917.928746:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534917.928757:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534917.928759:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534917.928759:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534917.928761:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534917.928761:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534917.928763:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534917.928764:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534917.928766:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584534917.988313:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584534917.988319:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584534917.988321:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584534917.988322:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584534917.988324:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584534917.988326:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584534917.988327:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584534917.988329:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584534998 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534917.988363:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078412432 new x1661492078412480 00000800:00020000:2.0:1584534923.916130:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534923.916133:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534923.916135:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534923.916545:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534923.928611:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534917/real 1584534917] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535004 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534923.928620:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534923.928621:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534923.928623:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534923.928639:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534923.928641:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534923.928642:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534923.928643:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534923.928644:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534923.928646:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534923.928646:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534923.928648:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16723 00000100:00080000:10.0:1584534923.928651:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534923.928657:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534923.928659:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534923.928659:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534923.928661:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534923.928663:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534923.928664:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534923.928665:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534923.928666:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584534924.046237:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584534924.046243:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584534924.046245:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584534924.046246:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584534924.046247:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584534924.046249:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584534924.046251:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584534924.046253:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535004 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534924.046287:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078412480 new x1661492078412528 00000800:00020000:2.0:1584534929.916063:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534929.916065:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534929.916067:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534929.916548:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534929.928604:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534924/real 1584534924] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535011 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534929.928612:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534929.928613:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534929.928614:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534929.928620:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534929.928622:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534929.928622:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534929.928624:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534929.928624:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534929.928626:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534929.928626:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534929.928628:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16729 00000100:00080000:10.0:1584534929.928630:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534929.928639:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534929.928640:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534929.928642:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534929.928643:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534929.928644:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534929.928645:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534929.928646:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534929.928647:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584534929.977983:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584534929.977987:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584534929.977989:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584534929.977989:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584534929.977991:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584534929.977993:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584534929.977994:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584534929.977996:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535011 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534929.978024:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078412528 new x1661492078412560 00000800:00020000:2.0:1584534935.916073:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534935.916075:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534935.916077:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534935.916693:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534935.928746:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534929/real 1584534929] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535016 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534935.928754:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534935.928755:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534935.928756:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534935.928762:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534935.928764:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534935.928774:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534935.928775:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534935.928776:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534935.928778:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534935.928778:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534935.928780:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16735 00000100:00080000:10.0:1584534935.928782:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534935.928789:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534935.928790:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534935.928791:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534935.928792:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534935.928792:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534935.928794:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534935.928794:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534935.928795:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584534935.977374:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584534935.977378:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584534935.977380:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584534935.977381:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584534935.977382:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584534935.977384:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584534935.977385:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584534935.977387:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535016 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534935.977417:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078412560 new x1661492078412624 00000800:00020000:4.0:1584534941.916154:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584534941.916156:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584534941.916158:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534941.916594:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534941.928646:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534935/real 1584534935] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535022 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534941.928653:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534941.928655:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534941.928656:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534941.928662:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534941.928663:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534941.928664:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534941.928665:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534941.928666:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534941.928667:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534941.928668:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534941.928670:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16741 00000100:00080000:10.0:1584534941.928672:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534941.928680:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534941.928682:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534941.928682:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534941.928683:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534941.928684:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534941.928685:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534941.928686:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534941.928687:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584534941.981792:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584534941.981796:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584534941.981798:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584534941.981799:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584534941.981800:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584534941.981802:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584534941.981803:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584534941.981805:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535022 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534941.981834:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078412624 new x1661492078412656 00000800:00020000:2.0:1584534947.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534947.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534947.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534947.916556:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534947.928608:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534941/real 1584534941] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535028 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534947.928615:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534947.928617:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534947.928618:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534947.928632:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534947.928634:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534947.928634:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534947.928636:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534947.928636:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534947.928638:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534947.928639:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534947.928641:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16747 00000100:00080000:10.0:1584534947.928643:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534947.928649:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534947.928650:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534947.928650:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534947.928651:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534947.928652:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534947.928653:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534947.928654:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534947.928655:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584534947.994309:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584534947.994313:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584534947.994315:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584534947.994316:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584534947.994317:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584534947.994319:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584534947.994321:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584534947.994322:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535028 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534947.994351:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078412656 new x1661492078412720 00000800:00020000:4.0:1584534953.916200:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584534953.916203:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584534953.916205:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534953.916665:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534953.928720:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534947/real 1584534947] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535034 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534953.928727:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534953.928728:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534953.928730:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534953.928735:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534953.928737:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534953.928737:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534953.928740:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534953.928740:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534953.928742:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534953.928742:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534953.928744:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16753 00000100:00080000:10.0:1584534953.928746:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534953.928754:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534953.928755:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534953.928756:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534953.928757:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534953.928758:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534953.928759:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534953.928759:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534953.928761:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584534953.984724:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584534953.984728:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584534953.984729:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584534953.984730:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584534953.984731:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584534953.984733:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584534953.984734:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584534953.984736:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535034 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534953.984766:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078412720 new x1661492078412752 00000800:00000100:0.0:1584534960.142721:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534960.143243:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534960.155311:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534953/real 1584534953] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535040 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534960.155318:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534960.155319:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534960.155321:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534960.155326:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534960.155328:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534960.155336:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534960.155337:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534960.155338:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534960.155339:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534960.155339:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534960.155341:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16759 00000100:00080000:10.0:1584534960.155343:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534960.155359:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534960.155360:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534960.155361:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534960.155362:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534960.155362:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534960.155364:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534960.155365:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534960.155366:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584534960.210910:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584534960.210914:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584534960.210917:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584534960.210918:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584534960.210919:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584534960.210921:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584534960.210922:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584534960.210924:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535040 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534960.210965:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078412752 new x1661492078412800 00000800:00020000:2.0:1584534965.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534965.938891:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534965.963066:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534965.963452:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534965.975534:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534960/real 1584534960] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535047 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534965.975542:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534965.975544:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534965.975546:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534965.975553:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534965.975555:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534965.975556:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534965.975557:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534965.975557:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534965.975559:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534965.975560:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534965.975562:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16766 00000100:00080000:10.0:1584534965.975565:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534965.975576:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534965.975577:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534965.975578:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534965.975579:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534965.975580:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534965.975581:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534965.975582:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534965.975583:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584534966.027923:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584534966.027929:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584534966.027931:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584534966.027932:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584534966.027934:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584534966.027936:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584534966.027937:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584534966.027939:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535047 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584534966.027968:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078412800 new x1661492078412848 00000800:00020000:2.0:1584534971.916071:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534971.916074:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (1): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534971.916076:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534971.916475:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584534971.928555:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534966/real 1584534966] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535053 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584534971.928569:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584534971.928571:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584534971.928573:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584534971.928591:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534971.928594:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534971.928595:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534971.928596:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534971.928597:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534971.928598:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584534971.928599:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584534971.928601:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16771 00000100:00080000:10.0:1584534971.928604:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584534971.928611:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534971.928612:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534971.928613:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534971.928614:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534971.928615:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534971.928616:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584534971.928617:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584534971.928618:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584534971.977267:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584534971.977273:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584534971.977275:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584534971.977276:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584534971.977278:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584534971.977280:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584534971.977281:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584534971.977283:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535053 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584534971.977313:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078412848 new x1661492078412896 00000800:00020000:4.0:1584534977.916042:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584534977.916045:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584534977.916048:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534977.916475:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584534977.928542:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534971/real 1584534971] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535058 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584534977.928551:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584534977.928552:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584534977.928554:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584534977.928561:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534977.928563:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534977.928564:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534977.928566:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534977.928567:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534977.928569:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584534977.928570:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584534977.928572:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16777 00000100:00080000:12.0:1584534977.928575:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584534977.928587:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534977.928588:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534977.928589:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534977.928590:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534977.928591:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534977.928592:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534977.928593:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534977.928594:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584534977.987763:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584534977.987769:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584534977.987771:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584534977.987772:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584534977.987774:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584534977.987777:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584534977.987778:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584534977.987780:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535058 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584534977.987809:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078412896 new x1661492078412944 00000800:00000100:0.0:1584534984.140707:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584534984.141138:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584534984.153209:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534977/real 1584534977] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535064 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584534984.153218:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584534984.153219:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584534984.153220:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584534984.153227:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534984.153229:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534984.153240:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534984.153242:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534984.153243:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534984.153245:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584534984.153246:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584534984.153247:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16783 00000100:00080000:12.0:1584534984.153250:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584534984.153256:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534984.153257:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534984.153258:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534984.153259:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534984.153260:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534984.153261:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584534984.153262:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584534984.153263:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584534984.204541:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584534984.204546:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584534984.204547:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584534984.204548:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584534984.204550:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584534984.204552:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584534984.204553:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584534984.204555:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535064 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584534984.204564:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078412944 new x1661492078412992 00000800:00020000:4.0:1584534989.916053:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584534989.916056:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584534989.916058:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534989.916559:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584534989.928611:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534984/real 1584534984] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535071 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584534989.928619:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584534989.928620:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584534989.928621:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584534989.928629:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534989.928631:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534989.928632:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534989.928633:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534989.928634:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534989.928635:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584534989.928636:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584534989.928637:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16790 00000100:00080000:14.0:1584534989.928639:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584534989.928650:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534989.928651:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534989.928652:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534989.928653:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534989.928653:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534989.928655:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534989.928655:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534989.928656:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534989.978739:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584534989.978743:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584534989.978744:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584534989.978745:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584534989.978747:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584534989.978748:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584534989.978750:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584534989.978752:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535071 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584534989.978782:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078412992 new x1661492078413024 00000800:00020000:2.0:1584534995.916067:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584534995.916070:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584534995.916072:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584534995.916457:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584534995.928509:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534989/real 1584534989] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535076 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584534995.928517:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584534995.928519:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584534995.928520:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584534995.928534:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534995.928536:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534995.928537:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534995.928538:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534995.928538:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534995.928540:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584534995.928540:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584534995.928542:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16795 00000100:00080000:14.0:1584534995.928544:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584534995.928550:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534995.928551:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534995.928552:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534995.928553:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534995.928553:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534995.928555:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584534995.928555:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584534995.928556:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584534995.982421:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584534995.982425:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584534995.982426:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584534995.982427:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584534995.982428:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584534995.982430:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584534995.982431:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584534995.982433:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535076 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584534995.982462:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078413024 new x1661492078413088 00000800:00020000:2.0:1584535001.916066:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535001.916068:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535001.916070:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535001.916486:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584535001.928539:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584534995/real 1584534995] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535082 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584535001.928547:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584535001.928548:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584535001.928549:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584535001.928555:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535001.928557:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535001.928557:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535001.928558:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535001.928559:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535001.928560:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584535001.928561:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584535001.928562:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16801 00000100:00080000:14.0:1584535001.928564:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584535001.928572:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535001.928574:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535001.928574:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535001.928575:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535001.928576:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535001.928577:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535001.928578:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535001.928579:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584535001.987201:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584535001.987205:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584535001.987206:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584535001.987207:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584535001.987209:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584535001.987210:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584535001.987212:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584535001.987214:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535082 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584535001.987222:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078413088 new x1661492078413120 00000800:00000100:0.0:1584535008.141641:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535008.142128:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584535008.154179:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535001/real 1584535001] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535088 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584535008.154187:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584535008.154188:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584535008.154189:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584535008.154196:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535008.154197:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535008.154206:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535008.154207:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535008.154208:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535008.154209:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584535008.154210:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584535008.154211:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16807 00000100:00080000:14.0:1584535008.154213:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584535008.154219:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535008.154221:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535008.154221:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535008.154222:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535008.154223:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535008.154224:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535008.154225:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535008.154227:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535008.205260:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584535008.205264:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584535008.205265:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584535008.205266:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584535008.205267:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584535008.205269:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584535008.205270:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584535008.205272:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535088 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584535008.205284:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078413120 new x1661492078413184 00000800:00020000:10.0:1584535013.916082:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:10.0:1584535013.916085:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:10.0:1584535013.916087:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535013.916499:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584535013.928553:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535008/real 1584535008] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535095 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584535013.928561:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584535013.928563:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584535013.928564:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584535013.928570:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535013.928572:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535013.928572:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535013.928574:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535013.928574:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535013.928576:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584535013.928576:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584535013.928578:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16814 00000100:00080000:16.0:1584535013.928580:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584535013.928592:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535013.928593:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535013.928593:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535013.928595:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535013.928595:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535013.928596:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535013.928597:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535013.928598:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535013.981171:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584535013.981175:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584535013.981177:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584535013.981178:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584535013.981179:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584535013.981181:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584535013.981183:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584535013.981184:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535095 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584535013.981192:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078413184 new x1661492078413216 00000800:00020000:2.0:1584535019.916046:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535019.916049:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535019.916050:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535019.916494:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584535019.928547:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535013/real 1584535013] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535100 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584535019.928554:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584535019.928555:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584535019.928557:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584535019.928570:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535019.928571:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535019.928572:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535019.928573:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535019.928574:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535019.928575:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584535019.928576:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584535019.928578:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16819 00000100:00080000:16.0:1584535019.928580:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584535019.928586:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535019.928587:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535019.928588:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535019.928589:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535019.928590:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535019.928591:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535019.928591:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535019.928592:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535019.995758:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584535019.995761:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584535019.995763:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584535019.995764:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584535019.995765:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584535019.995767:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584535019.995769:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584535019.995771:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535100 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584535019.995780:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078413216 new x1661492078413264 00000800:00020000:2.0:1584535025.916073:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535025.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535025.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535025.916499:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584535025.928547:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535019/real 1584535019] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535106 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584535025.928556:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584535025.928558:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584535025.928560:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584535025.928567:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535025.928569:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535025.928570:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535025.928571:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535025.928572:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535025.928574:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584535025.928575:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584535025.928577:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16825 00000100:00080000:16.0:1584535025.928580:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584535025.928593:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535025.928595:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535025.928595:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535025.928596:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535025.928597:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535025.928598:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535025.928599:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535025.928600:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535025.993042:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584535025.993048:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584535025.993050:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584535025.993051:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584535025.993053:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584535025.993055:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584535025.993057:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584535025.993059:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535106 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584535025.993088:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078413264 new x1661492078413312 00000800:00000100:0.0:1584535032.142603:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535032.143039:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584535032.155094:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535025/real 1584535025] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535112 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584535032.155104:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584535032.155106:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584535032.155108:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584535032.155116:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535032.155118:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535032.155133:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535032.155135:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535032.155136:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535032.155138:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584535032.155139:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584535032.155141:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16831 00000100:00080000:16.0:1584535032.155144:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584535032.155151:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535032.155153:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535032.155154:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535032.155155:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535032.155156:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535032.155157:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535032.155158:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535032.155159:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535032.212297:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584535032.212304:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584535032.212306:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584535032.212308:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584535032.212309:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584535032.212312:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584535032.212313:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584535032.212315:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535112 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584535032.212345:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078413312 new x1661492078413360 00000800:00020000:4.0:1584535037.916056:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535037.916060:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535037.916062:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535037.916475:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584535037.928532:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535032/real 1584535032] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535119 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584535037.928542:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584535037.928544:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584535037.928545:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584535037.928552:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535037.928555:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535037.928556:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535037.928557:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535037.928558:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535037.928560:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584535037.928561:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584535037.928563:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16838 00000100:00080000:16.0:1584535037.928566:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584535037.928581:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535037.928582:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535037.928583:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535037.928584:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535037.928585:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535037.928586:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535037.928587:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535037.928588:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535037.978412:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584535037.978419:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584535037.978421:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584535037.978423:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584535037.978424:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584535037.978427:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584535037.978429:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584535037.978431:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535119 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584535037.978442:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078413360 new x1661492078413408 00000800:00020000:2.0:1584535043.916066:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535043.916068:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535043.916071:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535043.916476:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584535043.928541:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535037/real 1584535037] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535124 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584535043.928550:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584535043.928552:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584535043.928554:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584535043.928575:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535043.928576:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535043.928577:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535043.928579:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535043.928580:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535043.928581:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584535043.928582:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584535043.928584:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16843 00000100:00080000:16.0:1584535043.928586:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584535043.928594:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535043.928595:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535043.928596:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535043.928597:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535043.928598:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535043.928599:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535043.928600:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535043.928601:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535043.988541:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584535043.988547:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584535043.988549:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584535043.988550:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584535043.988552:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584535043.988554:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584535043.988556:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584535043.988558:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535124 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584535043.988587:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078413408 new x1661492078413456 00000800:00020000:2.0:1584535049.916169:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535049.916171:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535049.916173:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535049.916616:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584535049.928669:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535043/real 1584535043] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535130 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584535049.928681:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584535049.928683:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584535049.928684:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584535049.928688:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535049.928690:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535049.928691:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535049.928692:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535049.928693:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535049.928694:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584535049.928695:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584535049.928697:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16849 00000100:00080000:16.0:1584535049.928699:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584535049.928709:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535049.928710:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535049.928711:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535049.928712:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535049.928713:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535049.928714:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535049.928715:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535049.928716:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535049.981734:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584535049.981738:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584535049.981740:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584535049.981741:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584535049.981742:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584535049.981744:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584535049.981746:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584535049.981747:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535130 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584535049.981755:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078413456 new x1661492078413488 00000800:00000100:0.0:1584535056.141466:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535056.141870:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584535056.153922:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535049/real 1584535049] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535136 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584535056.153930:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584535056.153931:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584535056.153932:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584535056.153938:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535056.153940:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535056.153949:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535056.153950:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535056.153951:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535056.153952:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584535056.153953:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584535056.153954:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16855 00000100:00080000:16.0:1584535056.153956:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584535056.153962:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535056.153964:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535056.153964:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535056.153965:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535056.153966:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535056.153967:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535056.153969:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535056.153970:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535056.177188:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535056.177192:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535056.177193:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535056.177194:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535056.177195:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535056.177197:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535056.177198:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535056.177200:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535136 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535056.177208:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078413488 new x1661492078413552 00000800:00020000:0.0:1584535061.916032:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584535061.916034:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584535061.916036:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535061.916634:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535061.928691:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535056/real 1584535056] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535143 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535061.928698:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535061.928700:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535061.928701:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535061.928706:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535061.928708:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535061.928709:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535061.928711:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535061.928711:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535061.928713:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535061.928713:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535061.928715:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16862 00000100:00080000:18.0:1584535061.928717:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535061.928726:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535061.928727:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535061.928728:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535061.928729:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535061.928730:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535061.928731:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535061.928731:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535061.928733:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535061.936476:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535061.936480:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535061.936482:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535061.936483:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535061.936484:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535061.936486:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535061.936487:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535061.936489:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535143 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535061.936497:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078413552 new x1661492078413584 00000800:00020000:4.0:1584535067.916147:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535067.916150:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535067.916151:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535067.916585:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535067.928642:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535061/real 1584535061] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535148 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535067.928650:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535067.928651:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535067.928653:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535067.928667:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535067.928669:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535067.928670:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535067.928671:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535067.928671:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535067.928673:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535067.928673:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535067.928675:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16867 00000100:00080000:18.0:1584535067.928677:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535067.928683:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535067.928684:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535067.928685:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535067.928686:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535067.928686:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535067.928688:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535067.928688:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535067.928689:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535067.936906:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535067.936910:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535067.936911:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535067.936912:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535067.936913:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535067.936915:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535067.936918:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535067.936920:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535148 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535067.936952:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078413584 new x1661492078413648 00000800:00020000:4.0:1584535073.916060:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535073.916062:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535073.916064:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535073.916476:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535073.928519:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535067/real 1584535067] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535154 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535073.928531:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535073.928533:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535073.928534:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535073.928539:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535073.928540:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535073.928541:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535073.928542:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535073.928543:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535073.928544:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535073.928545:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535073.928546:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16873 00000100:00080000:18.0:1584535073.928548:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535073.928557:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535073.928558:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535073.928559:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535073.928560:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535073.928560:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535073.928562:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535073.928562:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535073.928563:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535073.936253:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535073.936257:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535073.936258:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535073.936259:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535073.936260:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535073.936262:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535073.936263:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535073.936265:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535154 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535073.936295:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078413648 new x1661492078413680 00000800:00000100:0.0:1584535080.142376:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535080.143070:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535080.155123:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535073/real 1584535073] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535160 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535080.155131:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535080.155132:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535080.155133:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535080.155138:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535080.155140:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535080.155148:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535080.155149:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535080.155150:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535080.155151:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535080.155152:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535080.155153:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16879 00000100:00080000:18.0:1584535080.155155:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535080.155161:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535080.155162:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535080.155162:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535080.155164:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535080.155164:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535080.155165:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535080.155167:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535080.155168:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535080.163107:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535080.163111:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535080.163112:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535080.163113:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535080.163114:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535080.163116:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535080.163118:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535080.163119:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535160 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535080.163149:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078413680 new x1661492078413728 00000800:00020000:2.0:1584535085.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535085.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535085.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535085.916516:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535085.928567:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535080/real 1584535080] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535167 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535085.928575:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535085.928576:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535085.928577:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535085.928582:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535085.928583:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535085.928584:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535085.928586:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535085.928586:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535085.928588:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535085.928588:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535085.928590:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16886 00000100:00080000:18.0:1584535085.928592:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535085.928601:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535085.928602:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535085.928603:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535085.928604:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535085.928605:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535085.928607:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535085.928607:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535085.928608:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535085.936746:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535085.936750:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535085.936751:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535085.936752:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535085.936754:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535085.936756:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535085.936758:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535085.936760:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535167 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535085.936788:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078413728 new x1661492078413776 00000800:00020000:2.0:1584535091.916102:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535091.916105:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535091.916108:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535091.916731:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535091.928790:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535085/real 1584535085] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535172 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535091.928802:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535091.928803:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535091.928805:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535091.928829:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535091.928831:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535091.928832:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535091.928833:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535091.928834:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535091.928836:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535091.928837:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535091.928839:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16891 00000100:00080000:18.0:1584535091.928842:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535091.928850:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535091.928852:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535091.928853:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535091.928855:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535091.928855:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535091.928857:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535091.928857:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535091.928859:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535091.936925:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535091.936932:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535091.936934:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535091.936935:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535091.936937:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535091.936939:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535091.936941:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535091.936943:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535172 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535091.936971:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078413776 new x1661492078413824 00000800:00020000:2.0:1584535097.916063:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535097.916067:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535097.916069:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535097.916614:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535097.928667:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535091/real 1584535091] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535178 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535097.928676:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535097.928678:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535097.928680:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535097.928687:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535097.928689:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535097.928690:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535097.928691:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535097.928692:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535097.928693:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535097.928695:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535097.928697:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16897 00000100:00080000:18.0:1584535097.928699:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535097.928716:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535097.928717:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535097.928718:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535097.928719:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535097.928720:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535097.928721:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535097.928722:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535097.928723:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535097.936899:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535097.936905:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535097.936907:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535097.936908:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535097.936910:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535097.936913:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535097.936915:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535097.936917:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535178 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535097.936925:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078413824 new x1661492078413872 00000800:00000100:0.0:1584535104.140994:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535104.141391:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535104.153449:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535097/real 1584535097] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535184 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535104.153462:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535104.153463:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535104.153465:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535104.153475:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535104.153477:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535104.153494:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535104.153495:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535104.153496:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535104.153498:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535104.153499:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535104.153503:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16903 00000100:00080000:18.0:1584535104.153506:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535104.153515:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535104.153516:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535104.153517:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535104.153518:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535104.153519:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535104.153520:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535104.153521:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535104.153522:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535104.189423:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535104.189431:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535104.189433:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535104.189434:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535104.189436:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535104.189439:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535104.189441:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535104.189443:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535184 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535104.189474:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078413872 new x1661492078413920 00000800:00020000:2.0:1584535109.916063:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535109.916066:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535109.916068:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535109.916571:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535109.928623:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535104/real 1584535104] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535191 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535109.928631:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535109.928632:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535109.928634:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535109.928639:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535109.928641:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535109.928642:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535109.928643:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535109.928643:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535109.928646:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535109.928647:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535109.928648:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16910 00000100:00080000:18.0:1584535109.928650:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535109.928661:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535109.928662:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535109.928663:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535109.928664:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535109.928665:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535109.928666:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535109.928667:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535109.928668:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535109.936575:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535109.936580:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535109.936582:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535109.936583:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535109.936584:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535109.936586:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535109.936588:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535109.936589:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535191 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535109.936621:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078413920 new x1661492078413952 00000800:00020000:2.0:1584535115.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535115.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535115.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535115.916526:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535115.928578:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535109/real 1584535109] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535196 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535115.928586:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535115.928587:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535115.928588:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535115.928601:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535115.928603:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535115.928604:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535115.928606:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535115.928606:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535115.928608:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535115.928609:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535115.928610:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16915 00000100:00080000:18.0:1584535115.928612:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535115.928619:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535115.928620:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535115.928621:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535115.928622:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535115.928623:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535115.928624:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535115.928625:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535115.928626:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535115.936375:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535115.936380:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535115.936381:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535115.936382:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535115.936383:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535115.936385:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535115.936387:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535115.936389:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535196 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535115.936418:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078413952 new x1661492078414016 00000800:00020000:2.0:1584535121.916100:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535121.916102:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535121.916104:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535121.916513:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535121.928565:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535115/real 1584535115] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535202 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535121.928573:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535121.928574:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535121.928575:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535121.928581:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535121.928583:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535121.928584:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535121.928585:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535121.928586:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535121.928587:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535121.928588:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535121.928590:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16921 00000100:00080000:20.0:1584535121.928593:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535121.928602:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535121.928603:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535121.928604:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535121.928605:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535121.928605:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535121.928607:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535121.928607:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535121.928608:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535121.936910:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535121.936914:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535121.936915:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535121.936916:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535121.936918:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535121.936919:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535121.936922:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535121.936924:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535202 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535121.936953:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078414016 new x1661492078414048 00000800:00000100:0.0:1584535128.141653:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535128.142069:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535128.154120:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535121/real 1584535121] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535208 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535128.154128:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535128.154129:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535128.154130:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535128.154136:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535128.154138:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535128.154146:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535128.154147:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535128.154148:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535128.154150:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535128.154151:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535128.154153:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16927 00000100:00080000:20.0:1584535128.154155:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535128.154160:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535128.154162:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535128.154162:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535128.154163:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535128.154164:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535128.154165:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535128.154166:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535128.154167:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535128.162348:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535128.162352:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535128.162353:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535128.162354:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535128.162356:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535128.162358:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535128.162359:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535128.162361:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535208 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535128.162370:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078414048 new x1661492078414112 00000800:00020000:4.0:1584535133.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535133.916081:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535133.916083:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535133.916562:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535133.928615:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535128/real 1584535128] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535215 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535133.928624:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535133.928625:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535133.928626:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535133.928632:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535133.928633:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535133.928634:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535133.928635:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535133.928636:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535133.928637:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535133.928638:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535133.928639:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16934 00000100:00080000:22.0:1584535133.928641:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535133.928652:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535133.928654:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535133.928655:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535133.928656:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535133.928656:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535133.928658:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535133.928658:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535133.928659:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535133.936617:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535133.936621:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535133.936622:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535133.936623:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535133.936625:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535133.936627:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535133.936628:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535133.936630:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535215 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535133.936665:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078414112 new x1661492078414144 00000800:00020000:2.0:1584535139.916058:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535139.916060:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535139.916062:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535139.916549:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535139.928601:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535133/real 1584535133] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535220 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535139.928609:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535139.928610:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535139.928611:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535139.928624:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535139.928625:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535139.928626:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535139.928627:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535139.928628:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535139.928629:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535139.928630:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535139.928632:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16939 00000100:00080000:22.0:1584535139.928634:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535139.928640:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535139.928641:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535139.928642:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535139.928643:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535139.928643:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535139.928644:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535139.928645:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535139.928646:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535139.936555:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535139.936559:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535139.936560:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535139.936561:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535139.936563:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535139.936564:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535139.936567:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535139.936569:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535220 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535139.936600:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078414144 new x1661492078414192 00000800:00020000:4.0:1584535145.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535145.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535145.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535145.916468:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535145.928547:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535139/real 1584535139] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535226 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535145.928558:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535145.928559:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535145.928561:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535145.928568:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535145.928570:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535145.928571:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535145.928572:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535145.928573:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535145.928574:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535145.928576:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535145.928578:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16945 00000100:00080000:22.0:1584535145.928581:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535145.928593:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535145.928594:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535145.928595:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535145.928596:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535145.928597:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535145.928598:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535145.928599:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535145.928600:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535145.936359:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535145.936365:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535145.936367:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535145.936368:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535145.936370:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535145.936372:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535145.936373:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535145.936375:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535226 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535145.936404:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078414192 new x1661492078414240 00000800:00000100:0.0:1584535152.142522:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535152.142957:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535152.155010:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535145/real 1584535145] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535232 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535152.155019:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535152.155021:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535152.155022:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535152.155039:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535152.155041:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535152.155053:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535152.155055:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535152.155056:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535152.155058:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535152.155059:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535152.155071:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16951 00000100:00080000:22.0:1584535152.155074:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535152.155092:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535152.155094:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535152.155095:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535152.155096:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535152.155096:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535152.155098:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535152.155098:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535152.155100:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535152.163923:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535152.163929:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535152.163932:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535152.163933:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535152.163935:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535152.163937:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535152.163939:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535152.163941:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535232 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535152.163950:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078414240 new x1661492078414288 00000800:00020000:4.0:1584535157.916152:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535157.916156:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535157.916158:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535157.916582:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535157.928665:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535152/real 1584535152] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535239 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535157.928675:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535157.928676:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535157.928678:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535157.928686:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535157.928688:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535157.928690:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535157.928691:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535157.928692:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535157.928694:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535157.928695:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535157.928697:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16958 00000100:00080000:22.0:1584535157.928700:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535157.928713:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535157.928715:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535157.928715:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535157.928716:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535157.928717:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535157.928718:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535157.928719:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535157.928720:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535157.937244:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535157.937250:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535157.937253:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535157.937254:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535157.937256:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535157.937258:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535157.937261:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535157.937263:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535239 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535157.937293:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078414288 new x1661492078414336 00000800:00020000:2.0:1584535163.916112:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535163.916115:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535163.916117:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535163.916539:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535163.928592:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535157/real 1584535157] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535244 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535163.928601:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535163.928602:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535163.928604:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535163.928623:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535163.928625:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535163.928625:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535163.928627:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535163.928627:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535163.928629:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535163.928630:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535163.928632:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16963 00000100:00080000:22.0:1584535163.928634:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535163.928642:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535163.928643:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535163.928644:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535163.928645:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535163.928646:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535163.928647:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535163.928647:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535163.928649:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535163.954135:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535163.954141:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535163.954143:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535163.954144:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535163.954145:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535163.954148:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535163.954149:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535163.954151:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535244 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535163.954181:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078414336 new x1661492078414384 00000800:00020000:4.0:1584535169.916190:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535169.916193:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535169.916195:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535169.916737:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535169.928809:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535163/real 1584535163] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535250 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535169.928817:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535169.928819:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535169.928821:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535169.928827:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535169.928828:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535169.928829:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535169.928830:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535169.928831:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535169.928832:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535169.928833:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535169.928834:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16969 00000100:00080000:22.0:1584535169.928836:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535169.928845:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535169.928847:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535169.928847:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535169.928848:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535169.928849:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535169.928850:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535169.928851:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535169.928852:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535169.937423:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535169.937427:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535169.937429:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535169.937429:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535169.937431:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535169.937433:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535169.937434:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535169.937436:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535250 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535169.937465:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078414384 new x1661492078414416 00000800:00000100:0.0:1584535176.141511:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535176.141940:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535176.154009:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535169/real 1584535169] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535256 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535176.154018:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535176.154019:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535176.154021:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535176.154050:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535176.154052:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535176.154061:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535176.154063:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535176.154063:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535176.154065:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535176.154065:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535176.154067:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16975 00000100:00080000:22.0:1584535176.154069:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535176.154075:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535176.154076:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535176.154077:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535176.154078:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535176.154079:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535176.154080:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535176.154081:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535176.154082:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535176.162288:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535176.162292:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535176.162294:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535176.162295:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535176.162296:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535176.162298:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535176.162301:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535176.162302:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535256 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535176.162331:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078414416 new x1661492078414480 00000800:00020000:2.0:1584535181.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535181.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535181.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535181.916494:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535181.928544:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535176/real 1584535176] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535263 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535181.928552:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535181.928553:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535181.928554:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535181.928559:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535181.928561:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535181.928562:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535181.928563:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535181.928564:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535181.928565:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535181.928566:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535181.928567:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16982 00000100:00080000:22.0:1584535181.928569:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535181.928578:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535181.928579:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535181.928580:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535181.928581:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535181.928581:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535181.928583:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535181.928584:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535181.928585:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535181.992805:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535181.992809:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535181.992811:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535181.992811:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535181.992813:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535181.992814:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535181.992816:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535181.992818:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535263 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535181.992847:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078414480 new x1661492078414512 00000800:00020000:2.0:1584535187.916081:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535187.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535187.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535187.916490:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535187.928541:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535181/real 1584535181] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535268 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535187.928549:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535187.928550:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535187.928551:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535187.928565:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535187.928566:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535187.928567:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535187.928568:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535187.928569:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535187.928570:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535187.928571:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535187.928572:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16987 00000100:00080000:22.0:1584535187.928575:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535187.928581:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535187.928582:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535187.928582:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535187.928584:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535187.928585:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535187.928586:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535187.928587:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535187.928588:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535187.948589:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535187.948594:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535187.948595:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535187.948596:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535187.948597:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535187.948599:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535187.948601:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535187.948602:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535268 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535187.948632:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078414512 new x1661492078414576 00000800:00020000:6.0:1584535193.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584535193.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584535193.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535193.916527:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535193.928578:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535187/real 1584535187] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535274 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535193.928585:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535193.928587:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535193.928588:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535193.928593:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535193.928594:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535193.928595:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535193.928596:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535193.928597:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535193.928598:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535193.928599:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535193.928601:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16993 00000100:00080000:22.0:1584535193.928603:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535193.928611:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535193.928613:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535193.928614:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535193.928615:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535193.928616:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535193.928617:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535193.928617:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535193.928618:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535193.954542:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535193.954546:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535193.954547:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535193.954548:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535193.954549:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535193.954551:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535193.954554:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535193.954555:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535274 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535193.954585:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078414576 new x1661492078414608 00000800:00000100:0.0:1584535200.142324:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535200.142900:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535200.154975:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535193/real 1584535193] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535200.154983:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535200.154983:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535200.154985:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535200.154990:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535200.154992:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535200.154999:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535200.155000:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535200.155000:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535200.155001:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535200.155002:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535200.155003:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16999 00000100:00080000:22.0:1584535200.155016:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535200.155022:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535200.155023:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535200.155041:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535200.155042:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535200.155043:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535200.155055:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535200.155056:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535200.155057:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535200.206268:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535200.206276:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535200.206279:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535200.206281:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535200.206284:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535200.206288:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535200.206292:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535200.206296:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535200.206343:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078414608 new x1661492078414656 00000800:00020000:6.0:1584535205.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584535205.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584535205.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535205.916562:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535205.928642:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535200/real 1584535200] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535205.928654:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535205.928655:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535205.928657:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535205.928665:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535205.928668:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535205.928668:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535205.928670:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535205.928671:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535205.928672:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535205.928673:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535205.928676:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17006 00000100:00080000:22.0:1584535205.928680:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535205.928696:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535205.928697:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535205.928698:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535205.928699:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535205.928700:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535205.928701:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535205.928702:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535205.928703:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535205.969283:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535205.969290:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535205.969292:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535205.969293:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535205.969295:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535205.969298:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535205.969299:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535205.969301:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535205.969331:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078414656 new x1661492078414704 00000800:00020000:4.0:1584535211.916041:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535211.916045:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535211.916048:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535211.916497:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535211.928553:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535205/real 1584535205] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535211.928564:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535211.928566:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535211.928568:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535211.928593:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535211.928595:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535211.928596:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535211.928597:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535211.928598:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535211.928601:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535211.928602:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535211.928604:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17011 00000100:00080000:22.0:1584535211.928608:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535211.928616:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535211.928618:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535211.928618:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535211.928620:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535211.928621:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535211.928622:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535211.928623:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535211.928624:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535211.969321:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535211.969329:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535211.969331:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535211.969333:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535211.969334:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535211.969337:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535211.969340:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535211.969342:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535211.969371:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078414704 new x1661492078414752 00000800:00020000:4.0:1584535217.916148:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535217.916150:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535217.916152:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535217.916561:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535217.928612:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535211/real 1584535211] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535217.928620:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535217.928622:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535217.928623:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535217.928628:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535217.928630:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535217.928631:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535217.928633:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535217.928633:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535217.928635:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535217.928636:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535217.928637:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17017 00000100:00080000:22.0:1584535217.928640:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535217.928651:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535217.928652:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535217.928653:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535217.928654:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535217.928655:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535217.928656:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535217.928657:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535217.928658:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535217.969918:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535217.969924:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535217.969926:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535217.969927:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535217.969928:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535217.969930:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535217.969932:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535217.969934:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535217.969944:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078414752 new x1661492078414800 00000800:00000100:0.0:1584535224.141038:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535224.141456:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535224.153502:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535217/real 1584535217] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535224.153510:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535224.153511:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535224.153512:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535224.153517:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535224.153519:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535224.153528:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535224.153530:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535224.153530:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535224.153532:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535224.153532:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535224.153534:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17023 00000100:00080000:22.0:1584535224.153536:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535224.153542:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535224.153543:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535224.153544:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535224.153545:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535224.153546:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535224.153547:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535224.153547:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535224.153549:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535224.203629:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535224.203633:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535224.203635:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535224.203636:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535224.203637:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535224.203639:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535224.203640:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535224.203642:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535224.203672:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078414800 new x1661492078414848 00000800:00020000:4.0:1584535229.916102:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535229.916104:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535229.916106:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535229.916544:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535229.928601:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535224/real 1584535224] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535229.928609:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535229.928610:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535229.928612:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535229.928617:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535229.928619:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535229.928620:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535229.928621:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535229.928621:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535229.928623:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535229.928623:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535229.928625:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17030 00000100:00080000:22.0:1584535229.928627:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535229.928636:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535229.928637:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535229.928638:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535229.928639:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535229.928639:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535229.928641:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535229.928641:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535229.928642:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535229.982206:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535229.982210:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535229.982211:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535229.982212:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535229.982214:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535229.982215:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535229.982218:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535229.982220:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535229.982251:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078414848 new x1661492078414880 00000800:00020000:2.0:1584535235.916052:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535235.916055:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535235.916057:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535235.916459:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535235.928510:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535229/real 1584535229] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535235.928522:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535235.928523:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535235.928525:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535235.928538:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535235.928540:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535235.928541:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535235.928542:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535235.928543:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535235.928544:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535235.928544:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535235.928546:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17035 00000100:00080000:22.0:1584535235.928548:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535235.928554:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535235.928555:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535235.928556:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535235.928557:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535235.928558:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535235.928559:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535235.928559:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535235.928560:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535235.977780:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535235.977784:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535235.977786:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535235.977786:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535235.977788:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535235.977790:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535235.977791:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535235.977793:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535235.977822:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078414880 new x1661492078414944 00000800:00020000:2.0:1584535241.916058:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535241.916060:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535241.916062:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535241.916509:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535241.928560:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535235/real 1584535235] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535322 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535241.928568:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535241.928569:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535241.928570:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535241.928575:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535241.928577:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535241.928577:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535241.928578:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535241.928579:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535241.928580:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535241.928581:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535241.928583:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17041 00000100:00080000:22.0:1584535241.928585:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535241.928593:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535241.928594:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535241.928595:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535241.928596:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535241.928597:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535241.928598:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535241.928599:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535241.928600:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535241.967870:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535241.967874:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535241.967876:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535241.967876:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535241.967878:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535241.967879:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535241.967881:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535241.967883:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535322 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535241.967891:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078414944 new x1661492078414976 00000800:00000100:0.0:1584535248.141926:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535248.142384:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535248.154433:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535241/real 1584535241] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535328 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535248.154441:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535248.154442:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535248.154444:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535248.154448:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535248.154450:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535248.154459:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535248.154460:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535248.154461:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535248.154462:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535248.154463:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535248.154464:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17047 00000100:00080000:22.0:1584535248.154466:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535248.154472:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535248.154473:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535248.154474:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535248.154475:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535248.154476:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535248.154478:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535248.154478:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535248.154479:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535248.204672:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535248.204676:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535248.204677:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535248.204678:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535248.204679:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535248.204681:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535248.204684:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535248.204686:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535328 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535248.204715:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078414976 new x1661492078415040 00000800:00020000:6.0:1584535253.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584535253.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584535253.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535253.916494:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535253.928553:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535248/real 1584535248] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535335 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535253.928562:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535253.928563:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535253.928564:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535253.928570:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535253.928571:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535253.928572:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535253.928573:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535253.928574:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535253.928575:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535253.928576:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535253.928578:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17054 00000100:00080000:22.0:1584535253.928580:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535253.928589:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535253.928590:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535253.928591:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535253.928593:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535253.928593:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535253.928595:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535253.928595:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535253.928596:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535253.966246:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535253.966250:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535253.966252:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535253.966253:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535253.966254:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535253.966256:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535253.966258:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535253.966259:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535335 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535253.966288:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078415040 new x1661492078415072 00000800:00020000:2.0:1584535259.916115:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535259.916117:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535259.916119:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535259.916594:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535259.928645:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535253/real 1584535253] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535340 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535259.928652:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535259.928653:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535259.928655:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535259.928668:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535259.928669:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535259.928670:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535259.928671:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535259.928672:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535259.928673:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535259.928674:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535259.928675:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17059 00000100:00080000:22.0:1584535259.928677:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535259.928684:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535259.928685:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535259.928685:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535259.928687:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535259.928687:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535259.928688:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535259.928689:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535259.928690:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535259.975922:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535259.975926:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535259.975928:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535259.975929:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535259.975930:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535259.975932:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535259.975933:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535259.975935:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535340 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584535259.975945:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078415072 new x1661492078415120 00000800:00020000:2.0:1584535265.916047:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535265.916050:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535265.916052:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535265.916554:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584535265.928606:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535259/real 1584535259] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535346 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584535265.928614:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584535265.928615:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584535265.928618:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584535265.928623:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584535265.928625:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584535265.928626:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584535265.928627:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584535265.928628:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584535265.928629:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584535265.928630:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584535265.928632:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17065 00000100:00080000:4.0:1584535265.928634:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584535265.928643:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584535265.928645:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584535265.928645:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584535265.928647:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584535265.928647:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584535265.928649:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584535265.928649:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584535265.928650:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535265.966576:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535265.966581:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535265.966583:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535265.966584:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535265.966585:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535265.966587:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535265.966590:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535265.966592:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535346 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584535265.966601:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078415120 new x1661492078415168 00000800:00000100:0.0:1584535272.141813:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535272.142228:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584535272.154276:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535265/real 1584535265] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535352 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584535272.154293:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584535272.154294:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584535272.154296:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584535272.154304:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584535272.154306:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584535272.154322:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584535272.154323:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584535272.154324:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584535272.154326:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584535272.154327:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584535272.154329:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17071 00000100:00080000:4.0:1584535272.154333:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584535272.154340:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584535272.154342:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584535272.154342:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584535272.154344:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584535272.154344:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584535272.154346:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584535272.154347:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584535272.154348:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535272.207757:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535272.207765:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535272.207767:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535272.207769:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535272.207771:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535272.207773:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535272.207775:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535272.207777:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535352 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584535272.207787:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078415168 new x1661492078415216 00000800:00020000:2.0:1584535277.916155:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535277.916159:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535277.916161:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535277.916598:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584535277.928653:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535272/real 1584535272] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535359 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584535277.928662:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584535277.928664:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584535277.928666:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584535277.928672:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535277.928674:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535277.928675:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535277.928676:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535277.928677:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535277.928679:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584535277.928680:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584535277.928682:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17078 00000100:00080000:6.0:1584535277.928685:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584535277.928695:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535277.928697:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535277.928698:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535277.928700:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535277.928700:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535277.928702:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535277.928702:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535277.928704:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535277.970456:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535277.970463:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535277.970465:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535277.970467:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535277.970469:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535277.970472:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535277.970473:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535277.970475:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535359 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584535277.970483:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078415216 new x1661492078415264 00000800:00020000:2.0:1584535283.916121:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535283.916123:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535283.916125:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535283.916531:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584535283.928586:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535277/real 1584535277] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535364 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584535283.928594:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584535283.928596:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584535283.928597:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584535283.928611:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535283.928613:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535283.928614:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535283.928615:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535283.928616:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535283.928617:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584535283.928618:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584535283.928620:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17083 00000100:00080000:6.0:1584535283.928622:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584535283.928628:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535283.928631:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535283.928632:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535283.928633:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535283.928634:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535283.928635:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535283.928635:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535283.928637:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535283.984630:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535283.984636:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535283.984638:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535283.984639:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535283.984641:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535283.984643:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535283.984646:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535283.984647:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535364 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584535283.984676:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078415264 new x1661492078415312 00000800:00020000:2.0:1584535289.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535289.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535289.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535289.916488:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584535289.928562:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535283/real 1584535283] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535370 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584535289.928570:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584535289.928571:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584535289.928572:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584535289.928578:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535289.928580:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535289.928581:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535289.928582:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535289.928582:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535289.928584:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584535289.928584:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584535289.928586:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17089 00000100:00080000:6.0:1584535289.928589:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584535289.928597:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535289.928599:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535289.928599:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535289.928601:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535289.928601:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535289.928602:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535289.928603:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535289.928604:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535289.972765:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535289.972768:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535289.972770:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535289.972771:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535289.972772:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535289.972774:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535289.972775:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535289.972777:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535370 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584535289.972785:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078415312 new x1661492078415344 00000800:00000100:0.0:1584535296.141694:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535296.142115:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584535296.154165:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535289/real 1584535289] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535376 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584535296.154172:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584535296.154173:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584535296.154175:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584535296.154180:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535296.154181:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535296.154190:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535296.154191:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535296.154192:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535296.154193:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584535296.154194:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584535296.154196:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17095 00000100:00080000:6.0:1584535296.154198:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584535296.154204:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535296.154205:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535296.154205:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535296.154207:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535296.154207:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535296.154208:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535296.154209:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535296.154210:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535296.205357:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535296.205361:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535296.205362:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535296.205363:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535296.205365:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535296.205367:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535296.205368:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535296.205370:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535376 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584535296.205400:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078415344 new x1661492078415408 00000800:00020000:2.0:1584535301.916086:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535301.916088:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535301.916091:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535301.916513:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584535301.928566:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535296/real 1584535296] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535383 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584535301.928573:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584535301.928575:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584535301.928576:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584535301.928581:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535301.928582:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535301.928583:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535301.928584:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535301.928586:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535301.928587:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584535301.928588:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584535301.928589:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17102 00000100:00080000:6.0:1584535301.928591:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584535301.928600:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535301.928601:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535301.928601:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535301.928603:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535301.928603:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535301.928604:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535301.928605:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535301.928606:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535301.980456:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535301.980460:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535301.980461:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535301.980462:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535301.980464:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535301.980466:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535301.980468:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535301.980470:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535383 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584535301.980501:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078415408 new x1661492078415440 00000800:00020000:6.0:1584535307.916049:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584535307.916051:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584535307.916053:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535307.916431:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584535307.928485:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535301/real 1584535301] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535388 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584535307.928492:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584535307.928493:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584535307.928495:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584535307.928508:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535307.928510:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535307.928510:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535307.928512:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535307.928512:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535307.928514:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584535307.928514:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584535307.928516:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17107 00000100:00080000:6.0:1584535307.928518:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584535307.928523:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535307.928525:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535307.928525:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535307.928526:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535307.928527:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535307.928528:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535307.928529:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535307.928530:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535307.971174:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535307.971178:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535307.971180:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535307.971180:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535307.971182:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535307.971184:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535307.971185:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535307.971187:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535388 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584535307.971195:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078415440 new x1661492078415504 00000800:00020000:6.0:1584535313.916140:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584535313.916143:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584535313.916145:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535313.916561:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584535313.928639:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535307/real 1584535307] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535394 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584535313.928648:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584535313.928649:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584535313.928650:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584535313.928656:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535313.928657:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535313.928658:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535313.928659:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535313.928660:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535313.928661:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584535313.928662:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584535313.928663:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17113 00000100:00080000:6.0:1584535313.928666:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584535313.928674:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535313.928675:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535313.928676:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535313.928677:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535313.928678:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535313.928679:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535313.928680:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535313.928681:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535313.971746:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535313.971750:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535313.971751:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535313.971752:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535313.971753:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535313.971755:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535313.971757:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535313.971759:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535394 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584535313.971788:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078415504 new x1661492078415536 00000800:00000100:0.0:1584535320.142686:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535320.143246:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584535320.155325:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535313/real 1584535313] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535400 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584535320.155332:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584535320.155334:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584535320.155335:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584535320.155341:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535320.155342:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535320.155355:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535320.155356:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535320.155357:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535320.155359:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584535320.155359:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584535320.155361:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17119 00000100:00080000:6.0:1584535320.155363:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584535320.155368:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535320.155370:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535320.155370:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535320.155371:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535320.155372:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535320.155373:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535320.155375:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535320.155376:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535320.204990:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535320.204994:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535320.204996:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535320.204997:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535320.204998:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535320.205000:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535320.205003:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535320.205005:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535400 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584535320.205014:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078415536 new x1661492078415584 00000800:00020000:2.0:1584535325.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535325.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535325.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535325.916564:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584535325.928651:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535320/real 1584535320] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535407 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584535325.928660:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584535325.928661:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584535325.928663:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584535325.928670:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535325.928672:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535325.928673:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535325.928674:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535325.928675:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535325.928677:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584535325.928677:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584535325.928679:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17126 00000100:00080000:6.0:1584535325.928682:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584535325.928691:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535325.928692:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535325.928693:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535325.928694:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535325.928696:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535325.928698:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535325.928698:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535325.928699:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535325.986590:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535325.986596:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535325.986598:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535325.986599:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535325.986601:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535325.986603:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535325.986605:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535325.986607:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535407 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584535325.986617:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078415584 new x1661492078415632 00000800:00020000:4.0:1584535331.916041:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535331.916045:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535331.916047:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535331.916435:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584535331.928508:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535325/real 1584535325] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535412 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584535331.928519:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584535331.928520:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584535331.928522:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584535331.928547:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535331.928549:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535331.928550:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535331.928552:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535331.928552:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535331.928554:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584535331.928555:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584535331.928557:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17131 00000100:00080000:6.0:1584535331.928561:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584535331.928568:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535331.928570:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535331.928571:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535331.928573:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535331.928573:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535331.928575:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535331.928575:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535331.928577:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535331.974061:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535331.974068:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535331.974071:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535331.974072:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535331.974074:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535331.974076:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535331.974078:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535331.974080:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535412 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584535331.974109:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078415632 new x1661492078415680 00000800:00020000:4.0:1584535337.916174:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535337.916177:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535337.916179:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535337.916596:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584535337.928679:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535331/real 1584535331] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535418 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584535337.928688:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584535337.928689:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584535337.928690:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584535337.928696:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535337.928698:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535337.928699:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535337.928700:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535337.928701:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535337.928702:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584535337.928703:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584535337.928705:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17137 00000100:00080000:6.0:1584535337.928707:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584535337.928718:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535337.928720:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535337.928720:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535337.928722:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535337.928722:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535337.928723:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535337.928724:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535337.928725:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535337.978076:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535337.978081:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535337.978082:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535337.978083:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535337.978085:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535337.978087:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535337.978090:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535337.978092:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535418 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584535337.978102:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078415680 new x1661492078415728 00000800:00000100:0.0:1584535344.141093:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535344.141507:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584535344.153581:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535337/real 1584535337] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535424 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584535344.153589:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584535344.153590:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584535344.153592:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584535344.153597:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535344.153599:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535344.153607:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535344.153608:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535344.153609:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535344.153611:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584535344.153611:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584535344.153616:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17143 00000100:00080000:6.0:1584535344.153618:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584535344.153623:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535344.153625:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535344.153625:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535344.153626:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535344.153627:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535344.153628:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535344.153629:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535344.153630:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535344.200501:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535344.200505:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535344.200507:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535344.200508:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535344.200509:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535344.200511:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535344.200512:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535344.200514:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535424 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584535344.200543:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078415728 new x1661492078415776 00000800:00020000:2.0:1584535349.916058:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535349.916060:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535349.916062:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535349.916616:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584535349.928693:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535344/real 1584535344] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535431 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584535349.972467:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584535349.972468:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584535349.972470:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584535349.972474:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535349.972475:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535349.972476:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535349.972477:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535349.972478:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535349.972480:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584535349.972480:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584535349.972482:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17150 00000100:00080000:6.0:1584535349.972484:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584535349.972492:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535349.972494:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535349.972494:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535349.972495:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535349.972496:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535349.972497:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535349.972498:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535349.972499:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535350.025353:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535350.025357:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535350.025359:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535350.025359:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535350.025361:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535350.025362:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535350.025364:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535350.025366:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535431 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584535350.025395:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078415776 new x1661492078415808 00000800:00020000:2.0:1584535355.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535355.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535355.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535355.916525:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584535355.928594:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535350/real 1584535350] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535437 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584535355.928602:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584535355.928603:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584535355.928605:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584535355.928618:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535355.928620:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535355.928621:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535355.928623:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535355.928623:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535355.928625:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584535355.928625:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584535355.928627:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17155 00000100:00080000:6.0:1584535355.928629:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584535355.928634:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535355.928636:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535355.928636:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535355.928637:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535355.928638:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535355.928639:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535355.928640:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535355.928641:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535355.979851:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535355.979855:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535355.979857:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535355.979857:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535355.979859:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535355.979861:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535355.979864:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535355.979865:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535437 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584535355.979896:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078415808 new x1661492078415872 00000800:00020000:2.0:1584535361.916093:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535361.916095:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535361.916097:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535361.916514:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584535361.928596:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535355/real 1584535355] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535442 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584535361.928604:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584535361.928605:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584535361.928606:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584535361.928611:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535361.928614:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535361.928614:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535361.928615:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535361.928616:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535361.928617:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584535361.928618:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584535361.928620:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17161 00000100:00080000:6.0:1584535361.928622:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584535361.928630:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535361.928631:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535361.928632:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535361.928633:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535361.928634:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535361.928635:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535361.928635:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535361.928637:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535361.976862:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535361.976866:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535361.976868:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535361.976869:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535361.976870:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535361.976872:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535361.976873:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535361.976875:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535442 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584535361.976885:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078415872 new x1661492078415904 00000800:00000100:0.0:1584535368.141904:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535368.142495:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584535368.154575:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535361/real 1584535361] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535448 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584535368.154583:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584535368.154584:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584535368.154586:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584535368.154592:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535368.154594:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535368.154602:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535368.154604:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535368.154604:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535368.154606:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584535368.154606:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584535368.154608:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17167 00000100:00080000:6.0:1584535368.154610:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584535368.154615:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535368.154616:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535368.154617:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535368.154618:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535368.154619:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535368.154620:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535368.154621:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535368.154622:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535368.192918:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535368.192922:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535368.192924:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535368.192925:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535368.192926:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535368.192928:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535368.192929:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535368.192931:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535448 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584535368.192960:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078415904 new x1661492078415968 00000800:00020000:4.0:1584535373.916067:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535373.916069:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535373.916071:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535373.916507:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584535373.928584:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535368/real 1584535368] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535455 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584535373.928597:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584535373.928598:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584535373.928599:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584535373.928605:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535373.928606:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535373.928607:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535373.928608:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535373.928609:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535373.928610:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584535373.928611:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584535373.928612:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17174 00000100:00080000:6.0:1584535373.928614:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584535373.928623:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535373.928624:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535373.928625:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535373.928626:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535373.928627:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535373.928628:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535373.928629:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535373.928630:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535373.972174:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535373.972178:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535373.972179:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535373.972180:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535373.972182:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535373.972183:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535373.972186:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535373.972188:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535455 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584535373.972198:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078415968 new x1661492078416000 00000800:00020000:4.0:1584535379.916057:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535379.916059:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535379.916061:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535379.916580:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584535379.928659:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535373/real 1584535373] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535460 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584535379.928667:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584535379.928668:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584535379.928669:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584535379.928683:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535379.928684:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535379.928685:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535379.928686:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535379.928687:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535379.928688:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584535379.928689:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584535379.928691:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17179 00000100:00080000:6.0:1584535379.928693:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584535379.928698:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535379.928699:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535379.928700:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535379.928701:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535379.928701:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535379.928703:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535379.928704:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535379.928705:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535379.980711:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535379.980715:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535379.980716:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535379.980717:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535379.980718:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535379.980720:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535379.980722:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535379.980723:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535460 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584535379.980756:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078416000 new x1661492078416048 00000800:00020000:4.0:1584535385.916113:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535385.916117:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535385.916119:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535385.916514:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584535385.928595:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535379/real 1584535379] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535466 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584535385.928605:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584535385.928607:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584535385.928609:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584535385.928617:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535385.928619:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535385.928620:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535385.928622:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535385.928622:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535385.928624:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584535385.928625:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584535385.928627:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17185 00000100:00080000:6.0:1584535385.928631:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584535385.928646:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535385.928647:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535385.928648:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535385.928650:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535385.928651:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535385.928652:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535385.928653:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535385.928654:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535385.978855:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535385.978862:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535385.978864:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535385.978865:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535385.978867:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535385.978870:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535385.978871:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535385.978873:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535466 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584535385.978883:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078416048 new x1661492078416096 00000800:00000100:0.0:1584535392.141236:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535392.141701:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584535392.153769:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535385/real 1584535385] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535472 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584535392.153780:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584535392.153782:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584535392.153784:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584535392.153792:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535392.153794:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535392.153810:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535392.153811:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535392.153812:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535392.153814:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584535392.153815:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584535392.153817:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17191 00000100:00080000:6.0:1584535392.153820:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584535392.153828:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535392.153829:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535392.153831:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535392.153833:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535392.153833:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535392.153835:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535392.153835:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535392.153837:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535392.200786:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535392.200792:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535392.200795:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535392.200796:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535392.200798:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535392.200801:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535392.200803:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535392.200806:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535472 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584535392.200816:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078416096 new x1661492078416144 00000800:00020000:4.0:1584535397.916146:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535397.916149:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535397.916151:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535397.916533:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584535397.928585:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535392/real 1584535392] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535479 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584535397.928593:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584535397.928595:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584535397.928596:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584535397.928601:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535397.928603:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535397.928603:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535397.928605:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535397.928605:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535397.928607:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584535397.928607:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584535397.928609:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17198 00000100:00080000:6.0:1584535397.928611:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584535397.928623:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535397.928625:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535397.928625:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535397.928626:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535397.928627:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535397.928628:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535397.928629:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535397.928630:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535397.984385:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535397.984390:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535397.984392:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535397.984393:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535397.984395:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535397.984397:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535397.984398:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535397.984400:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535479 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584535397.984429:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078416144 new x1661492078416192 00000800:00020000:2.0:1584535403.916105:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535403.916108:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535403.916110:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535403.916523:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584535403.928600:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535397/real 1584535397] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535484 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584535403.928608:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584535403.928609:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584535403.928610:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584535403.928624:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535403.928626:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535403.928627:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535403.928628:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535403.928629:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535403.928630:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584535403.928631:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584535403.928633:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17203 00000100:00080000:6.0:1584535403.928635:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584535403.928640:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535403.928642:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535403.928642:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535403.928643:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535403.928644:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535403.928645:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584535403.928646:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584535403.928647:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535403.967659:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535403.967663:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535403.967665:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535403.967666:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535403.967667:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535403.967669:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535403.967671:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535403.967672:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535484 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584535403.967701:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078416192 new x1661492078416240 00000800:00020000:4.0:1584535409.916097:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535409.916100:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535409.916101:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535409.916634:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584535409.928704:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535403/real 1584535403] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535490 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584535409.928712:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584535409.928713:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584535409.928715:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584535409.928721:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584535409.928724:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584535409.928724:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584535409.928726:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584535409.928726:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584535409.928728:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584535409.928728:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584535409.928730:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17209 00000100:00080000:8.0:1584535409.928732:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584535409.928742:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584535409.928743:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584535409.928744:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584535409.928745:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584535409.928746:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584535409.928747:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584535409.928748:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584535409.928749:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535409.967735:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535409.967739:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535409.967740:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535409.967741:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535409.967742:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535409.967744:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535409.967747:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535409.967748:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535490 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584535409.967778:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078416240 new x1661492078416272 00000800:00000100:0.0:1584535416.141411:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535416.141893:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584535416.153961:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535409/real 1584535409] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535496 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584535416.153969:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584535416.153970:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584535416.153972:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584535416.153978:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584535416.153980:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584535416.153988:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584535416.153989:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584535416.153990:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584535416.153991:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584535416.153992:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584535416.153993:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17215 00000100:00080000:8.0:1584535416.153995:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584535416.154001:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584535416.154002:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584535416.154002:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584535416.154004:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584535416.154004:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584535416.154005:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584535416.154006:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584535416.154007:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535416.197639:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535416.197643:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535416.197644:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535416.197645:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535416.197647:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535416.197648:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535416.197650:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535416.197652:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535496 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584535416.197681:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078416272 new x1661492078416336 00000800:00020000:2.0:1584535421.916069:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535421.916071:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535421.916073:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535421.916482:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584535421.928558:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535416/real 1584535416] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535503 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584535421.928565:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584535421.953897:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584535421.953898:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584535421.953902:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584535421.953916:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535421.953917:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584535421.953918:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535421.953919:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584535421.953920:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584535421.953921:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584535421.953922:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17222 00000100:00080000:12.0:1584535421.953924:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584535421.953934:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584535421.953935:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535421.953936:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584535421.953937:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535421.953937:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584535421.953938:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535421.953939:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584535421.953940:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535422.881963:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584535422.881967:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584535422.881968:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584535422.881969:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584535422.881970:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584535422.881972:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584535422.881973:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584535422.899788:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535503 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584535422.899832:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078416336 new x1661492078416368 00000800:00020000:6.0:1584535428.916114:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584535428.916117:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584535428.916119:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535428.916502:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584535428.928557:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535422/real 1584535422] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535509 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584535428.928565:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584535428.928566:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584535428.928568:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584535428.928583:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535428.928584:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535428.928585:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535428.928586:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535428.928587:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535428.928588:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584535428.928589:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584535428.928591:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17227 00000100:00080000:14.0:1584535428.928593:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584535428.928598:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535428.928600:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535428.928600:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535428.928601:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535428.928602:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535428.928603:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535428.928604:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535428.928605:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535428.988954:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584535428.988959:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584535428.988960:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584535428.988961:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584535428.988962:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584535428.988964:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584535428.988966:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584535428.988967:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535509 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584535428.988997:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078416368 new x1661492078416432 00000800:00020000:2.0:1584535434.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535434.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535434.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535434.916522:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584535434.928612:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535428/real 1584535428] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535515 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584535434.928635:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584535434.928637:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584535434.928638:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584535434.928643:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535434.928644:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535434.928645:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535434.928646:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535434.928647:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535434.928648:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584535434.928649:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584535434.928650:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17234 00000100:00080000:14.0:1584535434.928652:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584535434.928660:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535434.928662:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535434.928662:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535434.928663:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535434.928664:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535434.928665:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535434.928666:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535434.928667:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535434.978220:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584535434.978224:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584535434.978225:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584535434.978226:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584535434.978227:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584535434.978229:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584535434.978230:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584535434.978232:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535515 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584535434.978279:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078416432 new x1661492078416464 00000800:00020000:6.0:1584535440.916169:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584535440.916172:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584535440.916175:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535440.916626:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584535440.928691:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535434/real 1584535434] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535521 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584535440.928702:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584535440.928704:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584535440.928706:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584535440.928713:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535440.928715:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535440.928730:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535440.928731:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535440.928732:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535440.928734:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584535440.928734:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584535440.928736:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17240 00000100:00080000:14.0:1584535440.928740:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584535440.928748:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535440.928749:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535440.928750:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535440.928751:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535440.928752:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535440.928753:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535440.928756:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535440.928757:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535440.979801:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584535440.979809:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584535440.979811:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584535440.979812:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584535440.979814:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584535440.979817:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584535440.979818:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584535440.979820:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535521 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584535440.979830:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078416464 new x1661492078416512 00000800:00020000:2.0:1584535446.916084:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535446.916087:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535446.916090:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535446.916567:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584535446.928690:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535440/real 1584535440] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535527 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584535446.928700:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584535446.928701:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584535446.928703:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584535446.928710:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535446.928712:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535446.928713:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535446.928715:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535446.928715:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535446.928717:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584535446.928718:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584535446.928720:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17246 00000100:00080000:14.0:1584535446.928723:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584535446.928736:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535446.928738:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535446.928738:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535446.928739:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535446.928741:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535446.928742:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535446.928743:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535446.928744:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535446.974189:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584535446.974196:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584535446.974198:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584535446.974199:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584535446.974201:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584535446.974204:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584535446.974205:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584535446.974207:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535527 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584535446.974240:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078416512 new x1661492078416560 00000800:00020000:6.0:1584535452.916081:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584535452.916084:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584535452.916086:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535452.916615:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584535452.928743:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535446/real 1584535446] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535533 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584535452.928753:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584535452.928754:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584535452.928755:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584535452.928774:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535452.928776:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535452.928777:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535452.928778:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535452.928779:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535452.928781:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584535452.928782:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584535452.928783:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17252 00000100:00080000:14.0:1584535452.928786:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584535452.928793:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535452.928795:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535452.928796:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535452.928798:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535452.928798:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535452.928800:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535452.928800:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535452.928801:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535452.979349:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584535452.979355:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584535452.979357:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584535452.979359:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584535452.979360:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584535452.979362:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584535452.979364:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584535452.979366:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535533 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584535452.979398:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078416560 new x1661492078416608 00000800:00020000:4.0:1584535458.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535458.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535458.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535458.916535:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584535458.928583:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535452/real 1584535452] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535539 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584535458.928592:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584535458.928594:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584535458.928595:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584535458.928602:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535458.928604:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535458.928605:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535458.928606:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535458.928607:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535458.928608:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584535458.928609:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584535458.928611:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17258 00000100:00080000:14.0:1584535458.928614:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584535458.928630:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535458.928632:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535458.928632:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535458.928634:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535458.928634:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535458.928636:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535458.928636:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535458.928637:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535458.984101:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584535458.984107:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584535458.984109:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584535458.984111:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584535458.984113:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584535458.984115:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584535458.984117:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584535458.984118:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535539 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584535458.984147:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078416608 new x1661492078416656 00000800:00000100:0.0:1584535465.140851:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535465.141289:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584535465.153339:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535458/real 1584535458] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535545 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584535465.153347:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584535465.153348:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584535465.153349:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584535465.153355:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535465.153356:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535465.153365:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535465.153367:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535465.153367:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535465.153369:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584535465.153369:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584535465.153372:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17264 00000100:00080000:14.0:1584535465.153374:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584535465.153379:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535465.153380:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535465.153381:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535465.153382:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535465.153383:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535465.153384:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535465.153384:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535465.153386:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535465.198001:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584535465.198006:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584535465.198007:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584535465.198008:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584535465.198009:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584535465.198011:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584535465.198013:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584535465.198014:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535545 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584535465.198045:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078416656 new x1661492078416704 00000800:00020000:2.0:1584535470.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535470.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535470.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535470.916512:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584535470.928566:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535465/real 1584535465] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535552 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584535470.928573:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584535470.928574:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584535470.928576:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584535470.928581:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535470.928583:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535470.928583:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535470.928585:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535470.928585:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535470.928588:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584535470.928588:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584535470.928590:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17271 00000100:00080000:14.0:1584535470.928592:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584535470.928600:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535470.928602:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535470.928602:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535470.928604:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535470.928604:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535470.928605:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535470.928606:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535470.928607:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535470.937271:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584535470.937275:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584535470.937276:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584535470.937277:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584535470.937279:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584535470.937281:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584535470.937282:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584535470.937284:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535552 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584535470.937312:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078416704 new x1661492078416736 00000800:00020000:2.0:1584535476.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535476.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535476.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535476.916493:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584535476.928545:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535470/real 1584535470] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535557 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584535476.928553:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584535476.928554:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584535476.928555:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584535476.928569:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535476.928571:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535476.928572:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535476.928574:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535476.928574:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535476.928576:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584535476.928576:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584535476.928578:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17276 00000100:00080000:14.0:1584535476.928580:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584535476.928586:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535476.928587:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535476.928587:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535476.928589:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535476.928589:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535476.928590:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535476.928591:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535476.928592:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535476.936245:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584535476.936249:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584535476.936251:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584535476.936252:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584535476.936253:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584535476.936255:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584535476.936256:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584535476.936258:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535557 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584535476.936286:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078416736 new x1661492078416800 00000800:00020000:4.0:1584535482.916083:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535482.916086:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535482.916087:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535482.916535:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584535482.928592:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535476/real 1584535476] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535563 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584535482.928600:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584535482.928601:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584535482.928602:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584535482.928607:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535482.928609:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535482.928610:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535482.928611:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535482.928612:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535482.928613:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584535482.928614:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584535482.928615:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17282 00000100:00080000:14.0:1584535482.928617:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584535482.928626:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535482.928627:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535482.928628:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535482.928629:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535482.928629:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535482.928631:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535482.928631:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535482.928632:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535482.936944:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584535482.936948:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584535482.936949:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584535482.936950:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584535482.936951:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584535482.936953:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584535482.936954:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584535482.936956:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535563 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584535482.936985:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078416800 new x1661492078416832 00000800:00000100:0.0:1584535489.141407:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535489.141831:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584535489.154021:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535482/real 1584535482] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535569 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584535489.154041:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584535489.154042:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584535489.154049:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584535489.154055:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535489.154056:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535489.154066:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535489.154067:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535489.154068:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535489.154069:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584535489.154070:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584535489.154072:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17288 00000100:00080000:14.0:1584535489.154074:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584535489.154080:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535489.154082:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535489.154082:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535489.154083:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535489.154084:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535489.154085:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535489.154086:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535489.154087:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535489.161915:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584535489.161920:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584535489.161922:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584535489.161923:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584535489.161924:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584535489.161926:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584535489.161928:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584535489.161930:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535569 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584535489.161959:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078416832 new x1661492078416896 00000800:00020000:2.0:1584535494.916057:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535494.916060:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535494.916062:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535494.916508:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584535494.928560:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535489/real 1584535489] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535576 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584535494.928572:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584535494.928573:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584535494.928575:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584535494.928580:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535494.928581:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535494.928582:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535494.928583:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535494.928584:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535494.928585:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584535494.928586:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584535494.928587:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17295 00000100:00080000:14.0:1584535494.928589:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584535494.928598:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535494.928599:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535494.928600:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535494.928601:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535494.928602:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535494.928603:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535494.928603:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535494.928604:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535494.936410:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584535494.936414:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584535494.936416:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584535494.936417:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584535494.936418:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584535494.936420:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584535494.936422:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584535494.936423:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535576 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584535494.936452:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078416896 new x1661492078416928 00000800:00020000:2.0:1584535500.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535500.916081:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535500.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535500.916529:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584535500.928585:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535494/real 1584535494] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535581 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584535500.928593:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584535500.928594:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584535500.928595:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584535500.928609:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535500.928610:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535500.928611:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535500.928612:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535500.928613:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535500.928614:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584535500.928615:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584535500.928617:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17300 00000100:00080000:14.0:1584535500.928618:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584535500.928624:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535500.928625:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535500.928626:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535500.928627:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535500.928628:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535500.928629:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535500.928630:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535500.928631:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535500.936515:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535500.936519:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535500.936521:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535500.936522:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535500.936523:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535500.936525:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535500.936527:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535500.936528:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535581 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584535500.936557:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078416928 new x1661492078416976 00000800:00020000:2.0:1584535506.916059:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535506.916062:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535506.916064:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535506.916474:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584535506.928529:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535500/real 1584535500] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535587 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584535506.928538:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584535506.928539:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584535506.928540:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584535506.928546:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535506.928548:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535506.928548:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535506.928550:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535506.928550:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535506.928552:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584535506.928552:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584535506.928554:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17306 00000100:00080000:14.0:1584535506.928556:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584535506.928565:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535506.928566:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535506.928567:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535506.928568:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535506.928570:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535506.928571:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535506.928571:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535506.928572:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535506.936440:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535506.936444:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535506.936446:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535506.936447:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535506.936448:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535506.936450:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535506.936452:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535506.936454:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535587 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584535506.936484:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078416976 new x1661492078417024 00000800:00000100:0.0:1584535513.142060:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535513.142467:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584535513.154551:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535506/real 1584535506] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535593 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584535513.154563:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584535513.154564:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584535513.154566:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584535513.154575:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535513.154577:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535513.154592:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535513.154593:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535513.154594:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535513.154596:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584535513.154597:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584535513.154599:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17312 00000100:00080000:14.0:1584535513.154603:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584535513.154611:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535513.154612:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535513.154614:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535513.154615:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535513.154616:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535513.154617:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535513.154618:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535513.154619:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535513.183359:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535513.183366:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535513.183368:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535513.183370:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535513.183371:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535513.183374:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535513.183375:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535513.183377:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535593 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584535513.183386:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078417024 new x1661492078417072 00000800:00020000:4.0:1584535518.916073:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535518.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535518.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535518.916533:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584535518.928649:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535513/real 1584535513] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535600 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584535518.928660:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584535518.928662:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584535518.928663:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584535518.928671:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535518.928673:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535518.928674:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535518.928675:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535518.928676:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535518.928678:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584535518.928679:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584535518.928681:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17319 00000100:00080000:14.0:1584535518.928684:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584535518.928699:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535518.928701:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535518.928702:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535518.928703:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535518.928704:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535518.928705:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535518.928706:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535518.928707:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535518.936618:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535518.936625:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535518.936628:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535518.936629:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535518.936631:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535518.936634:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535518.936635:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535518.936637:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535600 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584535518.936666:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078417072 new x1661492078417120 00000800:00020000:4.0:1584535524.916122:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535524.916124:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535524.916126:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535524.916635:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584535524.928689:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535518/real 1584535518] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535605 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584535524.928699:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584535524.928700:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584535524.928702:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584535524.928722:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535524.928724:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535524.928725:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535524.928726:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535524.928727:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535524.928728:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584535524.928729:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584535524.928732:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17324 00000100:00080000:14.0:1584535524.928734:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584535524.928741:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535524.928743:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535524.928743:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535524.928745:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535524.928746:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535524.928747:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535524.928748:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535524.928749:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535524.936368:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535524.936375:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535524.936377:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535524.936378:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535524.936380:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535524.936383:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535524.936385:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535524.936387:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535605 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584535524.936396:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078417120 new x1661492078417168 00000800:00020000:2.0:1584535530.916052:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535530.916054:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535530.916056:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535530.916460:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584535530.928514:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535524/real 1584535524] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535611 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584535530.928522:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584535530.928523:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584535530.928524:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584535530.928530:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535530.928531:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535530.928532:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535530.928533:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535530.928534:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535530.928536:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584535530.928537:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584535530.928539:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17330 00000100:00080000:14.0:1584535530.928541:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584535530.928550:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535530.928551:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535530.928552:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535530.928553:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535530.928553:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535530.928555:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535530.928555:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535530.928556:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535530.936420:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535530.936424:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535530.936425:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535530.936426:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535530.936427:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535530.936429:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535530.936431:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535530.936432:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535611 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584535530.936461:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078417168 new x1661492078417200 00000800:00000100:0.0:1584535537.140198:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535537.140633:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584535537.152683:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535530/real 1584535530] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535617 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584535537.152691:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584535537.152692:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584535537.152693:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584535537.152699:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535537.152701:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535537.152709:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535537.152712:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535537.152712:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535537.152714:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584535537.152714:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584535537.152716:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17336 00000100:00080000:14.0:1584535537.152718:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584535537.152724:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535537.152725:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535537.152725:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535537.152727:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535537.152727:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535537.152728:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535537.152729:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535537.152730:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535537.160479:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535537.160483:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535537.160484:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535537.160485:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535537.160486:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535537.160488:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535537.160490:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535537.160492:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535617 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584535537.160500:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078417200 new x1661492078417264 00000800:00020000:2.0:1584535542.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535542.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535542.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535542.916553:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584535542.928641:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535537/real 1584535537] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535624 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584535542.928659:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584535542.928661:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584535542.928662:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584535542.928667:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535542.928670:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535542.928671:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535542.928672:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535542.928673:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535542.928674:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584535542.928675:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584535542.928676:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17343 00000100:00080000:14.0:1584535542.928678:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584535542.928686:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535542.928688:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535542.928688:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535542.928689:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535542.928690:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535542.928691:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535542.928692:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535542.928693:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535542.936787:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535542.936791:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535542.936793:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535542.936793:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535542.936795:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535542.936797:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535542.936799:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535542.936801:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535624 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584535542.936850:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078417264 new x1661492078417296 00000800:00020000:4.0:1584535548.916109:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535548.916111:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535548.916113:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535548.916526:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584535548.928580:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535542/real 1584535542] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535629 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584535548.928588:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584535548.928589:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584535548.928591:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584535548.928605:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535548.928606:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535548.928607:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535548.928608:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535548.928609:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535548.928610:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584535548.928611:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584535548.928612:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17348 00000100:00080000:14.0:1584535548.928615:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584535548.928620:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535548.928622:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535548.928622:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535548.928624:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535548.928624:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535548.928625:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584535548.928626:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584535548.928627:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535548.936506:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535548.936510:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535548.936511:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535548.936512:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535548.936513:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535548.936515:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535548.936517:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535548.936518:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535629 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584535548.936548:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078417296 new x1661492078417360 00000800:00020000:2.0:1584535554.916045:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535554.916047:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535554.916049:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535554.916528:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584535554.928611:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535548/real 1584535548] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535635 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584535554.928619:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584535554.928620:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584535554.928622:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584535554.928628:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535554.928630:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535554.928631:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535554.928632:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535554.928632:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535554.928634:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584535554.928635:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584535554.928636:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17354 00000100:00080000:16.0:1584535554.928638:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584535554.928648:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535554.928649:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535554.928650:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535554.928651:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535554.928651:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535554.928653:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535554.928653:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535554.928654:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535554.936385:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535554.936390:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535554.936391:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535554.936392:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535554.936393:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535554.936395:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535554.936397:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535554.936398:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535635 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584535554.936427:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078417360 new x1661492078417392 00000800:00000100:0.0:1584535561.142159:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535561.142576:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584535561.154628:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535554/real 1584535554] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535641 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584535561.154640:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584535561.154642:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584535561.154643:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584535561.154649:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535561.154651:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535561.154659:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535561.154661:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535561.154661:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535561.154663:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584535561.154663:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584535561.154665:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17360 00000100:00080000:16.0:1584535561.154668:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584535561.154673:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535561.154674:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535561.154675:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535561.154676:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535561.154677:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535561.154678:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584535561.154679:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584535561.154680:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535561.162415:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535561.162419:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535561.162420:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535561.162421:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535561.162422:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535561.162424:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535561.162427:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535561.162429:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535641 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535561.162438:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078417392 new x1661492078417440 00000800:00020000:2.0:1584535566.916059:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535566.938868:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535566.963040:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535566.963476:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535566.975564:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535561/real 1584535561] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535648 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535566.975586:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535566.975588:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535566.975590:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535566.975598:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535566.975600:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535566.975601:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535566.975602:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535566.975603:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535566.975604:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535566.975605:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535566.975607:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17367 00000100:00080000:18.0:1584535566.975610:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535566.975625:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535566.975627:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535566.975628:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535566.975629:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535566.975630:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535566.975631:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535566.975633:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535566.975634:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535566.983431:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535566.983437:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535566.983439:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535566.983440:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535566.983442:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535566.983445:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535566.983446:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535566.983448:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535648 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535566.983483:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078417440 new x1661492078417488 00000800:00020000:4.0:1584535572.916062:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535572.916065:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535572.916067:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535572.916457:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535572.928557:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535566/real 1584535566] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535653 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535572.928567:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535572.928568:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535572.928570:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535572.928592:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535572.928594:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535572.928595:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535572.928596:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535572.928597:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535572.928610:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535572.928611:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535572.928613:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17372 00000100:00080000:18.0:1584535572.928615:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535572.928623:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535572.928624:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535572.928625:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535572.928626:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535572.928628:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535572.928630:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535572.928630:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535572.928632:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535572.936369:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535572.936376:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535572.936378:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535572.936379:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535572.936381:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535572.936384:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535572.936385:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535572.936387:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535653 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535572.936421:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078417488 new x1661492078417536 00000800:00020000:2.0:1584535578.916102:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535578.916108:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535578.916113:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535578.916631:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535578.928724:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535572/real 1584535572] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535659 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535578.928744:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535578.928746:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535578.928748:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535578.928755:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535578.928757:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535578.928758:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535578.928759:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535578.928760:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535578.928762:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535578.928763:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535578.928765:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17378 00000100:00080000:18.0:1584535578.928767:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535578.928780:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535578.928781:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535578.928783:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535578.928784:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535578.928785:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535578.928786:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535578.928787:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535578.928788:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535578.936805:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535578.936812:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535578.936814:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535578.936815:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535578.936817:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535578.936819:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535578.936822:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535578.936824:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535659 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535578.936854:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078417536 new x1661492078417584 00000800:00000100:0.0:1584535585.141377:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535585.141792:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535585.153846:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535578/real 1584535578] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535665 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535585.153855:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535585.153856:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535585.153858:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535585.153865:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535585.153867:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535585.153878:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535585.153880:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535585.153881:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535585.153882:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535585.153883:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535585.153885:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17384 00000100:00080000:18.0:1584535585.153887:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535585.153896:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535585.153897:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535585.153898:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535585.153899:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535585.153900:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535585.153901:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535585.153902:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535585.153903:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535585.182035:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535585.182041:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535585.182043:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535585.182044:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535585.182046:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535585.182048:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535585.182050:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535585.182052:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535665 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535585.182061:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078417584 new x1661492078417632 00000800:00020000:2.0:1584535590.916097:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535590.916102:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535590.916107:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535590.916655:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535590.928749:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535585/real 1584535585] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535672 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535590.928768:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535590.928769:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535590.928770:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535590.928777:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535590.928778:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535590.928779:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535590.928780:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535590.928781:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535590.928782:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535590.928783:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535590.928785:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17391 00000100:00080000:18.0:1584535590.928787:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535590.928795:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535590.928797:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535590.928798:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535590.928799:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535590.928800:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535590.928801:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535590.928801:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535590.928802:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535590.937064:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535590.937068:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535590.937070:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535590.937070:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535590.937072:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535590.937074:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535590.937075:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535590.937077:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535672 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535590.937106:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078417632 new x1661492078417664 00000800:00020000:2.0:1584535596.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535596.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535596.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535596.916548:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535596.928632:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535590/real 1584535590] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535677 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535596.928640:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535596.928641:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535596.928642:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535596.928657:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535596.928659:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535596.928660:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535596.928661:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535596.928662:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535596.928664:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535596.928665:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535596.928667:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17396 00000100:00080000:18.0:1584535596.928669:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535596.928674:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535596.928676:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535596.928677:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535596.928678:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535596.928678:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535596.928679:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535596.928680:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535596.928681:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535596.936583:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535596.936587:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535596.936589:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535596.936590:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535596.936591:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535596.936593:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535596.936596:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535596.936597:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535677 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535596.936626:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078417664 new x1661492078417728 00000800:00020000:4.0:1584535602.916103:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535602.916108:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535602.916113:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535602.916574:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535602.928660:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535596/real 1584535596] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535683 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535602.928667:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535602.928679:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535602.928681:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535602.928686:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535602.928688:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535602.928688:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535602.928691:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535602.928691:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535602.928693:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535602.928693:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535602.928695:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17402 00000100:00080000:18.0:1584535602.928697:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535602.928705:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535602.928706:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535602.928707:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535602.928708:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535602.928708:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535602.928710:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535602.928710:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535602.928711:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535602.968692:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535602.968700:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535602.968703:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535602.968705:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535602.968708:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535602.968712:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535602.968715:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535602.968719:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535683 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535602.968779:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078417728 new x1661492078417760 00000800:00000100:0.0:1584535609.142040:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535609.142557:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535609.154615:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535602/real 1584535602] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535689 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535609.154623:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535609.154624:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535609.154625:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535609.154631:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535609.154634:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535609.154642:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535609.154644:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535609.154644:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535609.154646:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535609.154646:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535609.154648:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17408 00000100:00080000:18.0:1584535609.154650:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535609.154656:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535609.154657:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535609.154658:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535609.154659:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535609.154659:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535609.154660:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535609.154661:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535609.154662:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535609.181720:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535609.181724:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535609.181725:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535609.181726:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535609.181728:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535609.181730:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535609.181731:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535609.181733:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535689 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535609.181745:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078417760 new x1661492078417824 00000800:00020000:4.0:1584535614.916100:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535614.916106:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535614.916110:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535614.916575:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535614.928651:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535609/real 1584535609] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535696 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535614.928669:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535614.928670:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535614.928673:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535614.928678:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535614.928680:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535614.928681:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535614.928682:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535614.928682:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535614.928684:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535614.928684:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535614.928686:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17415 00000100:00080000:18.0:1584535614.928687:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535614.928696:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535614.928697:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535614.928697:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535614.928699:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535614.928699:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535614.928700:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535614.928701:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535614.928702:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535614.959275:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535614.959284:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535614.959287:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535614.959288:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535614.959291:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535614.959296:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535614.959301:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535614.959305:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535696 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535614.959353:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078417824 new x1661492078417856 00000800:00020000:2.0:1584535620.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535620.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535620.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535620.916516:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535620.928569:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535614/real 1584535614] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535701 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535620.928581:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535620.928582:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535620.928584:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535620.928597:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535620.928599:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535620.928600:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535620.928601:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535620.928601:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535620.928603:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535620.928603:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535620.928605:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17420 00000100:00080000:18.0:1584535620.928607:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535620.928612:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535620.928614:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535620.928614:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535620.928615:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535620.928616:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535620.928617:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535620.928618:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535620.928619:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535620.951450:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535620.951454:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535620.951455:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535620.951456:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535620.951458:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535620.951459:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535620.951461:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535620.951462:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535701 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535620.951491:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078417856 new x1661492078417904 00000800:00020000:2.0:1584535626.916102:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535626.916107:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535626.916112:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535626.916659:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535626.928749:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535620/real 1584535620] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535707 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535626.928768:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535626.928769:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535626.928770:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535626.928776:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535626.928778:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535626.928778:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535626.928779:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535626.928780:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535626.928782:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535626.928782:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535626.928784:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17426 00000100:00080000:18.0:1584535626.928786:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535626.928795:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535626.928796:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535626.928797:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535626.928798:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535626.928798:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535626.928799:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535626.928801:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535626.928813:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535626.937788:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535626.937792:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535626.937794:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535626.937794:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535626.937796:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535626.937798:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535626.937799:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535626.937801:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535707 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535626.937850:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078417904 new x1661492078417952 00000800:00000100:0.0:1584535633.142630:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535633.143081:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535633.155174:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535626/real 1584535626] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535713 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535633.155184:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535633.155185:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535633.155187:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535633.155195:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535633.155197:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535633.155209:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535633.155211:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535633.155211:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535633.155213:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535633.155214:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535633.155216:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17432 00000100:00080000:18.0:1584535633.155219:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535633.155226:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535633.155227:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535633.155228:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535633.155229:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535633.155231:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535633.155232:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535633.155233:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535633.155234:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535633.163719:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535633.163725:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535633.163727:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535633.163728:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535633.163730:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535633.163732:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535633.163735:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535633.163737:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535713 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535633.163768:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078417952 new x1661492078418000 00000800:00020000:10.0:1584535638.916123:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:10.0:1584535638.916129:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:10.0:1584535638.916134:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535638.916576:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535638.928698:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535633/real 1584535633] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535720 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535638.928708:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535638.928710:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535638.928712:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535638.928720:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535638.928722:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535638.928723:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535638.928725:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535638.928725:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535638.928727:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535638.928728:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535638.928730:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17439 00000100:00080000:18.0:1584535638.928733:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535638.928748:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535638.928749:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535638.928751:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535638.928753:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535638.928753:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535638.928754:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535638.928755:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535638.928756:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535638.965066:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535638.965077:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535638.965081:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535638.965083:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535638.965087:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535638.965092:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535638.965095:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535638.965099:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535720 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535638.965145:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078418000 new x1661492078418048 00000800:00020000:2.0:1584535644.916041:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535644.916045:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535644.916047:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535644.916477:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535644.928529:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535638/real 1584535638] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535725 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535644.928540:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535644.928541:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535644.928543:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535644.928565:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535644.928567:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535644.928568:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535644.928569:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535644.928570:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535644.928572:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535644.928573:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535644.928575:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17444 00000100:00080000:18.0:1584535644.928578:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535644.928587:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535644.928589:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535644.928590:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535644.928591:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535644.928592:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535644.928593:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535644.928593:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535644.928595:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535644.937311:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535644.937318:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535644.937320:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535644.937322:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535644.937324:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535644.937326:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535644.937328:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535644.937330:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535725 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535644.937361:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078418048 new x1661492078418096 00000800:00020000:2.0:1584535650.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535650.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (2): c: 6, oc: 0, rc: 8 00000800:00000100:2.0:1584535650.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535650.916573:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535650.928657:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535644/real 1584535644] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535731 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535650.928676:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535650.928677:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535650.928678:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535650.928684:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535650.928686:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535650.928687:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535650.928688:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535650.928689:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535650.928690:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535650.928691:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535650.928694:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17450 00000100:00080000:18.0:1584535650.928696:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535650.928704:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535650.928705:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535650.928706:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535650.928707:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535650.928708:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535650.928709:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535650.928710:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535650.928711:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535650.937311:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535650.937316:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535650.937318:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535650.937319:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535650.937321:0:25724:0:(client.c:2816:ptlrpc_resend_req()) @@@ going to resend req@ffff8d2babe41f80 x1661492078418112/t0(0) o17->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 456/432 e 0 to 0 dl 1584535691 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000100:00080000:22.0:1584535650.937331:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535650.937333:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535650.937335:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535650.937346:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535731 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535650.937394:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078418096 new x1661492078418160 00000800:00000100:0.0:1584535657.140626:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535657.141092:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535657.153145:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535650/real 1584535650] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535737 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535657.153154:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535657.153155:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535657.153157:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535657.153164:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535657.153166:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535657.153167:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535657.153168:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535657.153168:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535657.153171:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535657.153172:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535657.153174:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17456 00000100:00080000:18.0:1584535657.153177:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535657.153187:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535657.153189:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535657.153189:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535657.153191:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535657.153192:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535657.153193:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535657.153194:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535657.153195:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535657.185310:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535657.185316:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535657.185318:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535657.185319:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535657.185320:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535657.185322:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535657.185324:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535657.185326:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535737 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535657.185355:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078418160 new x1661492078418208 00000800:00020000:4.0:1584535662.916120:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535662.916126:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535662.916130:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535662.916541:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535662.928624:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535657/real 1584535657] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535744 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535662.928632:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535662.928633:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535662.928645:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535662.928651:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535662.928652:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535662.928653:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535662.928655:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535662.928656:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535662.928657:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535662.928658:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535662.928659:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17463 00000100:00080000:18.0:1584535662.928661:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535662.928670:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535662.928671:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535662.928671:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535662.928673:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535662.928673:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535662.928674:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535662.928675:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535662.928676:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535662.937514:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535662.937518:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535662.937519:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535662.937520:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535662.937521:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535662.937523:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535662.937525:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535662.937526:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535744 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535662.937562:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078418208 new x1661492078418240 00000800:00020000:2.0:1584535668.916056:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535668.916058:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535668.916060:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535668.916467:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535668.928520:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535662/real 1584535662] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535749 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535668.928528:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535668.928529:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535668.928530:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535668.928545:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535668.928548:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535668.928549:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535668.928550:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535668.928551:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535668.928552:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535668.928553:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535668.928554:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17468 00000100:00080000:18.0:1584535668.928557:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535668.928563:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535668.928564:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535668.928564:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535668.928566:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535668.928566:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535668.928568:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535668.928568:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535668.928569:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535668.937142:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535668.937146:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535668.937147:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535668.937149:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535668.937150:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535668.937152:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535668.937154:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535668.937155:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535749 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584535668.937184:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078418240 new x1661492078418304 00000800:00020000:0.0:1584535674.916068:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584535674.916071:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584535674.916073:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535674.916547:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584535674.928596:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535668/real 1584535668] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535755 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584535674.928604:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584535674.928605:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584535674.928607:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584535674.928613:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535674.928615:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535674.928624:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535674.928625:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535674.928626:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535674.928627:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584535674.928628:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584535674.928629:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17474 00000100:00080000:18.0:1584535674.928631:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584535674.928637:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535674.928638:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535674.928639:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535674.928640:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535674.928641:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535674.928642:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584535674.928643:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584535674.928644:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535674.969521:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535674.969525:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535674.969526:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535674.969527:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535674.969528:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535674.969530:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535674.969532:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535674.969533:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535755 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535674.969542:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078418304 new x1661492078418352 00000800:00000100:0.0:1584535681.141186:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535681.141617:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535681.153674:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535674/real 1584535674] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535761 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535681.153682:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535681.153683:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535681.153684:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535681.153691:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535681.153693:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535681.153693:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535681.153695:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535681.153695:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535681.153697:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535681.153697:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535681.153699:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17480 00000100:00080000:20.0:1584535681.153701:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535681.153711:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535681.153715:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535681.153715:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535681.153717:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535681.153717:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535681.153718:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535681.153719:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535681.153720:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535681.179521:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535681.179525:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535681.179526:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535681.179527:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535681.179529:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535681.179530:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535681.179532:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535681.179534:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535761 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535681.179562:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078418352 new x1661492078418384 00000800:00020000:2.0:1584535686.916061:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535686.916065:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535686.916067:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535686.916518:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535686.928601:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535681/real 1584535681] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535768 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535686.928611:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535686.928613:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535686.928614:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535686.928621:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535686.928623:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535686.928624:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535686.928626:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535686.928626:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535686.928628:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535686.928629:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535686.928631:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17487 00000100:00080000:20.0:1584535686.928635:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535686.928647:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535686.928648:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535686.928649:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535686.928650:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535686.928651:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535686.928652:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535686.928653:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535686.928654:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535686.974076:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535686.974082:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535686.974085:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535686.974086:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535686.974088:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535686.974090:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535686.974092:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535686.974094:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535768 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535686.974123:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078418384 new x1661492078418432 00000100:00000400:0.0:1584535691.153100:0:25725:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584535648/real 1584535648] req@ffff8d2babe40900 x1661492078418128/t0(0) o17->public1-OST0001-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 456/432 e 0 to 1 dl 1584535691 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 00000100:02000400:0.0:1584535691.153114:0:25725:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection to public1-OST0001 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:0.0:1584535691.153116:0:25725:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from FULL to DISCONN 00000100:00080000:0.0:1584535691.153118:0:25725:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0001_UUID: waking up pinger s:DISCONN 00000100:00080000:0.0:1584535691.153146:0:166:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1584535691.153148:0:166:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1584535691.153149:0:166:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1584535691.153151:0:166:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:0.0:1584535691.153152:0:166:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1584535691.153154:0:166:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 16372 00000100:00080000:0.0:1584535691.153158:0:166:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:0.0:1584535691.153166:0:166:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1584535691.153167:0:166:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1584535691.153168:0:166:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1584535691.153170:0:166:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1584535691.153171:0:166:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1584535691.153172:0:166:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1584535691.153172:0:166:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1584535691.153174:0:166:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1584535691.153175:0:166:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1584535691.153177:0:166:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000800:00020000:4.0:1584535692.916066:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535692.916069:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (1): c: 7, oc: 0, rc: 8 00000800:00000100:4.0:1584535692.916072:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535692.916518:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535692.928610:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535686/real 1584535686] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535773 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535692.928619:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535692.928632:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535692.928633:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535692.928640:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535692.928642:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535692.928643:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535692.928644:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535692.928645:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535692.928646:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535692.928648:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535692.928650:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17492 00000100:00080000:20.0:1584535692.928653:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535692.928666:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535692.928667:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535692.928668:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535692.928670:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535692.928670:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535692.928671:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535692.928672:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535692.928673:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535692.952621:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584535692.952628:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584535692.952631:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584535692.952632:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584535692.952634:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584535692.952637:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584535692.952638:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584535692.952640:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535773 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535692.952670:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078418432 new x1661492078418496 00000100:00080000:10.0:1584535697.928069:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:10.0:1584535697.928071:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0001_UUID on 10.10.2.22@o2ib failed (-110) 00000800:00020000:8.0:1584535698.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584535698.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584535698.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535698.916469:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535698.928523:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535692/real 1584535692] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535779 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535698.928532:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535698.928533:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535698.928535:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535698.928541:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535698.928543:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535698.928544:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535698.928545:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535698.928547:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535698.928548:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535698.928549:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535698.928551:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17498 00000100:00080000:20.0:1584535698.928553:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535698.928564:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535698.928565:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535698.928566:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535698.928567:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535698.928568:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535698.928569:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535698.928569:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535698.928571:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584535698.966361:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584535698.966367:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584535698.966369:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584535698.966370:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584535698.966372:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584535698.966374:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584535698.966376:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584535698.966378:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535779 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535698.966387:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078418496 new x1661492078418544 00000800:00000100:0.0:1584535705.141232:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535705.141659:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535705.153712:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535698/real 1584535698] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535785 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535705.153721:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535705.153722:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535705.153723:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535705.153729:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535705.153731:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535705.153733:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535705.153735:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535705.153735:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535705.153737:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535705.153738:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535705.153739:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17504 00000100:00080000:20.0:1584535705.153742:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535705.153753:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535705.153754:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535705.153755:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535705.153756:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535705.153757:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535705.153758:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535705.153759:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535705.153760:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584535705.196352:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584535705.196357:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584535705.196359:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584535705.196361:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584535705.196362:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584535705.196365:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584535705.196366:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584535705.196368:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535785 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535705.196398:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078418544 new x1661492078418576 00000800:00020000:6.0:1584535710.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584535710.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584535710.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535710.916590:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535710.928641:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535705/real 1584535705] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535792 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535710.928648:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535710.928649:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535710.928651:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535710.928658:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535710.928659:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535710.928660:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535710.928661:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535710.928662:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535710.928663:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535710.928664:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535710.928666:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17511 00000100:00080000:20.0:1584535710.928668:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535710.928676:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535710.928677:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535710.928678:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535710.928679:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535710.928680:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535710.928681:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535710.928681:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535710.928683:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584535710.970130:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584535710.970135:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584535710.970136:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584535710.970137:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584535710.970139:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584535710.970140:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584535710.970142:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584535710.970144:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535792 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535710.970152:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078418576 new x1661492078418608 00000800:00020000:8.0:1584535716.916041:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584535716.916044:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584535716.916046:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535716.916522:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535716.928572:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535710/real 1584535710] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535797 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535716.928581:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535716.928582:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535716.928584:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535716.928599:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535716.928600:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535716.928601:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:20.0:1584535716.928603:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:20.0:1584535716.928603:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535716.928605:0:1231:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17497 00000100:00080000:20.0:1584535716.928607:0:1231:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 16347 00000100:00080000:20.0:1584535716.928609:0:1231:0:(import.c:596:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0001 (at 10.10.2.21@o2ib) 00000100:00080000:20.0:1584535716.928610:0:1231:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.21@o2ib/10.10.2.21@o2ib 00000100:00080000:20.0:1584535716.928616:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535716.928617:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535716.928618:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535716.928619:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535716.928619:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535716.928621:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17516 00000100:00080000:20.0:1584535716.928622:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535716.928625:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535716.928626:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535716.928627:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535716.928628:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535716.928629:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535716.928631:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535716.928631:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535716.928632:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584535716.928918:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:10.0:1584535716.928920:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0001_UUID on 10.10.2.21@o2ib failed (-19) 00000100:00080000:10.0:1584535716.977143:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584535716.977147:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584535716.977149:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584535716.977149:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584535716.977151:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584535716.977152:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584535716.977154:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584535716.977156:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535797 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535716.977185:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078418608 new x1661492078418688 00000800:00020000:2.0:1584535722.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535722.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535722.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535722.916488:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535722.928539:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535716/real 1584535716] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535803 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535722.928547:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535722.928548:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535722.928549:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535722.928555:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535722.928556:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535722.928557:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535722.928559:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535722.928559:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535722.928561:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535722.928561:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535722.928563:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17522 00000100:00080000:20.0:1584535722.928565:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535722.928573:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535722.928575:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535722.928576:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535722.928577:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535722.928578:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535722.928579:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535722.928580:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535722.928581:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584535722.969959:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584535722.969963:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584535722.969964:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584535722.969965:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584535722.969966:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584535722.969968:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584535722.969969:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584535722.969971:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535803 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535722.969980:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078418688 new x1661492078418720 00000800:00000100:0.0:1584535729.141905:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535729.142341:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535729.154391:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535722/real 1584535722] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535809 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535729.154399:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535729.154400:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535729.154402:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535729.154407:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535729.154409:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535729.154410:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535729.154411:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535729.154412:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535729.154413:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535729.154414:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535729.154415:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17528 00000100:00080000:20.0:1584535729.154417:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535729.154427:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535729.154429:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535729.154429:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535729.154430:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535729.154431:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535729.154432:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535729.154433:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535729.154434:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584535729.201356:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584535729.201360:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584535729.201362:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584535729.201363:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584535729.201364:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584535729.201366:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584535729.201367:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584535729.201369:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535809 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535729.201399:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078418720 new x1661492078418768 00000800:00020000:8.0:1584535734.916064:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584535734.916067:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584535734.916069:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535734.916506:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535734.928557:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535729/real 1584535729] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535816 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535734.928565:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535734.928566:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535734.928567:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535734.928572:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535734.928574:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535734.928575:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535734.928576:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535734.928577:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535734.928578:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535734.928579:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535734.928581:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17535 00000100:00080000:20.0:1584535734.928583:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535734.928592:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535734.928593:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535734.928594:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535734.928595:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535734.928596:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535734.928597:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535734.928597:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535734.928599:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584535734.969850:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584535734.969854:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584535734.969856:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584535734.969857:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584535734.969858:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584535734.969860:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584535734.969862:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584535734.969863:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535816 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535734.969872:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078418768 new x1661492078418800 00000800:00020000:2.0:1584535740.916061:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535740.916063:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535740.916065:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535740.916478:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535740.928529:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535734/real 1584535734] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535821 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535740.928537:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535740.928538:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535740.928539:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535740.928554:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535740.928556:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535740.928557:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:20.0:1584535740.928558:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:20.0:1584535740.928559:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535740.928562:0:1231:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17497 00000100:00080000:20.0:1584535740.928564:0:1231:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 17522 00000100:00080000:20.0:1584535740.928565:0:1231:0:(import.c:570:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: tried all connections, increasing latency to 6s 00000100:00080000:20.0:1584535740.928567:0:1231:0:(import.c:596:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0001 (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584535740.928568:0:1231:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535740.928574:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535740.928575:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535740.928576:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535740.928577:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535740.928577:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535740.928578:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17540 00000100:00080000:20.0:1584535740.928580:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535740.928583:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535740.928584:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535740.928585:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535740.928586:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535740.928587:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535740.928588:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535740.928588:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535740.928590:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584535740.936460:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584535740.936464:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584535740.936466:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584535740.936467:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584535740.936468:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584535740.936470:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584535740.936471:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584535740.936473:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535821 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584535740.936482:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000100:00080000:20.0:1584535740.936483:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078418800 new x1661492078418864 00000080:00080000:10.0:1584535740.936484:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584535740.936484:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0001_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584535740.936485:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584535740.936486:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584535740.936487:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584535740.936488:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584535740.936489:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727f80): req@ffff8d2babe40900 x1661492078418128/t0(0) o17->public1-OST0001-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 456/432 e 0 to 1 dl 1584535691 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 00000800:00020000:2.0:1584535746.916061:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535746.916064:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535746.916066:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535746.916478:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535746.928530:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535740/real 1584535740] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535827 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535746.928538:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535746.928539:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535746.928540:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535746.928546:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535746.928548:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535746.928549:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535746.928550:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535746.928550:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535746.928552:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535746.928552:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535746.928554:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17546 00000100:00080000:20.0:1584535746.928556:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535746.928567:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535746.928568:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535746.928569:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535746.928570:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535746.928570:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535746.928572:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535746.928572:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535746.928574:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584535746.945941:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584535746.945945:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584535746.945947:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584535746.945948:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584535746.945949:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584535746.945951:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584535746.945953:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584535746.945954:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535827 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535746.945963:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078418864 new x1661492078418912 00000800:00000100:0.0:1584535753.142301:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535753.142769:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535753.154877:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535746/real 1584535746] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535833 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535753.154887:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535753.154889:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535753.154890:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535753.154897:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535753.154899:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535753.154900:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535753.154901:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535753.154902:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535753.154903:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535753.154904:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535753.154906:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17552 00000100:00080000:20.0:1584535753.154908:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535753.154919:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535753.154920:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535753.154921:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535753.154922:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535753.154923:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535753.154925:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535753.154926:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535753.154927:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584535753.162767:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584535753.162773:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584535753.162774:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584535753.162775:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584535753.162777:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584535753.162779:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584535753.162780:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584535753.162782:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535833 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535753.162814:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078418912 new x1661492078418944 00000800:00020000:2.0:1584535758.916059:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535758.916062:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535758.916065:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535758.916454:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535758.928511:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535753/real 1584535753] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535840 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535758.928520:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535758.928522:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535758.928524:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535758.928531:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535758.928533:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535758.928534:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535758.928535:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535758.928536:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535758.928538:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535758.928538:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535758.928541:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17559 00000100:00080000:20.0:1584535758.928543:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535758.928555:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535758.928556:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535758.928557:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535758.928559:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535758.928560:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535758.928561:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535758.928562:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535758.928563:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584535758.979344:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584535758.979351:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584535758.979353:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584535758.979354:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584535758.979356:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584535758.979358:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584535758.979360:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584535758.979362:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535840 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535758.979373:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078418944 new x1661492078418992 00000800:00020000:2.0:1584535764.916059:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535764.916061:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535764.916063:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535764.916479:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535764.928532:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535758/real 1584535758] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535845 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535764.928541:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535764.928542:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535764.928543:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535764.928557:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535764.928559:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535764.928564:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535764.928565:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535764.928566:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535764.928567:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535764.928568:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535764.928570:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17564 00000100:00080000:20.0:1584535764.928572:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535764.928577:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535764.928581:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535764.928581:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535764.928582:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535764.928583:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535764.928584:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535764.928585:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535764.928586:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584535764.973490:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584535764.973494:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584535764.973495:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584535764.973496:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584535764.973498:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584535764.973500:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584535764.973501:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584535764.973503:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535845 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535764.973512:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078418992 new x1661492078419056 00000800:00020000:2.0:1584535770.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535770.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535770.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535770.916535:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535770.928625:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535764/real 1584535764] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535851 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535770.928633:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535770.928634:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535770.928635:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535770.928641:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535770.928643:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535770.928644:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535770.928645:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535770.928646:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535770.928647:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535770.928648:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535770.928649:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17570 00000100:00080000:20.0:1584535770.928652:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535770.928660:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535770.928662:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535770.928662:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535770.928664:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535770.928664:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535770.928665:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535770.928666:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535770.928667:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584535770.948534:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584535770.948538:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584535770.948539:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584535770.948540:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584535770.948541:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584535770.948543:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584535770.948544:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584535770.948546:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535851 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535770.948577:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078419056 new x1661492078419088 00000800:00000100:0.0:1584535777.141274:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535777.141785:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535777.153860:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535770/real 1584535770] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535857 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535777.153871:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535777.153873:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535777.153875:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535777.153883:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535777.153884:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535777.153885:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535777.153887:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535777.153887:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535777.153889:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535777.153891:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535777.153893:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17576 00000100:00080000:20.0:1584535777.153896:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535777.153910:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535777.153911:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535777.153912:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535777.153914:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535777.153914:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535777.153915:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535777.153916:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535777.153928:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584535777.193070:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584535777.193077:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584535777.193079:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584535777.193080:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584535777.193082:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584535777.193085:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584535777.193086:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584535777.193088:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535857 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535777.193121:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078419088 new x1661492078419136 00000800:00020000:4.0:1584535782.916092:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535782.916094:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535782.916096:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535782.916502:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535782.928585:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535777/real 1584535777] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535864 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535782.928603:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535782.928604:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535782.928606:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535782.928611:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535782.928613:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535782.928614:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535782.928615:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535782.928616:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535782.928618:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535782.928618:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535782.928620:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17583 00000100:00080000:20.0:1584535782.928622:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535782.928631:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535782.928632:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535782.928633:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535782.928634:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535782.928634:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535782.928636:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535782.928636:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535782.928637:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584535782.965342:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584535782.965358:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584535782.965360:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584535782.965361:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584535782.965362:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584535782.965364:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584535782.965365:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584535782.965367:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535864 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535782.965401:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078419136 new x1661492078419168 00000800:00020000:2.0:1584535788.916068:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535788.916070:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535788.916072:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535788.916482:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535788.928538:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535782/real 1584535782] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535869 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535788.928547:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535788.928548:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535788.928549:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535788.928565:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535788.928567:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535788.928572:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535788.928573:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535788.928574:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535788.928575:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535788.928576:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535788.928577:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17588 00000100:00080000:20.0:1584535788.928579:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535788.928586:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535788.928587:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535788.928588:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535788.928589:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535788.928589:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535788.928591:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535788.928591:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535788.928592:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584535788.962577:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584535788.962582:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584535788.962584:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584535788.962585:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584535788.962586:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584535788.962588:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584535788.962590:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584535788.962591:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535869 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535788.962621:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078419168 new x1661492078419248 00000800:00020000:4.0:1584535794.916144:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535794.916149:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535794.916154:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535794.916578:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535794.928670:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535788/real 1584535788] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535875 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535794.928688:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535794.928689:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535794.928691:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535794.928697:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535794.928699:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535794.928699:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535794.928700:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535794.928701:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535794.928702:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535794.928703:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535794.928704:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17594 00000100:00080000:20.0:1584535794.928706:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535794.928714:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535794.928716:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535794.928716:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535794.928717:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535794.928718:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535794.928719:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535794.928720:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535794.928721:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584535794.987319:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584535794.987323:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584535794.987324:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584535794.987325:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584535794.987327:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584535794.987329:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584535794.987330:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584535794.987332:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535875 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535794.987362:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078419248 new x1661492078419280 00000800:00000100:0.0:1584535801.142942:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535801.143377:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535801.155428:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535794/real 1584535794] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535881 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535801.155436:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535801.155438:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535801.155440:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535801.155445:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535801.155447:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535801.155448:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535801.155449:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535801.155450:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535801.155451:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535801.155451:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535801.155453:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17600 00000100:00080000:20.0:1584535801.155455:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535801.155463:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535801.155465:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535801.155465:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535801.155466:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535801.155467:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535801.155468:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535801.155469:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535801.155470:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584535801.203203:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584535801.203207:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584535801.203208:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584535801.203209:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584535801.203211:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584535801.203213:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584535801.203214:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584535801.203216:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535881 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535801.203224:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078419280 new x1661492078419312 00000800:00020000:2.0:1584535806.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535806.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535806.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535806.916518:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535806.928617:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535801/real 1584535801] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535888 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535806.928638:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535806.928639:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535806.928641:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535806.928649:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535806.928651:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535806.928652:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535806.928653:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535806.928654:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535806.928655:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535806.928656:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535806.928658:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17607 00000100:00080000:20.0:1584535806.928661:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535806.928673:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535806.928674:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535806.928675:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535806.928676:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535806.928677:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535806.928678:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535806.928679:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535806.928680:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584535806.973335:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584535806.973341:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584535806.973343:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584535806.973344:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584535806.973346:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584535806.973348:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584535806.973349:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584535806.973351:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535888 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535806.973382:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078419312 new x1661492078419360 00000800:00020000:2.0:1584535812.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535812.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535812.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535812.916508:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535812.928573:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535806/real 1584535806] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535893 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535812.928582:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535812.928583:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535812.928584:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535812.928615:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535812.928617:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535812.928621:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535812.928622:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535812.928623:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535812.928624:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535812.928625:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535812.928626:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17612 00000100:00080000:20.0:1584535812.928628:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535812.928634:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535812.928636:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535812.928636:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535812.928638:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535812.928638:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535812.928640:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535812.928641:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535812.928642:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584535812.981372:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584535812.981377:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584535812.981379:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584535812.981380:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584535812.981381:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584535812.981384:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584535812.981385:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584535812.981387:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535893 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535812.981416:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078419360 new x1661492078419424 00000800:00020000:0.0:1584535818.916036:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584535818.916039:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584535818.916041:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535818.916654:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535818.928786:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535812/real 1584535812] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535899 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535818.928797:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535818.928798:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535818.928800:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535818.928808:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535818.928810:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535818.928811:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535818.928812:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535818.928813:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535818.928814:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535818.928815:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535818.928817:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17618 00000100:00080000:20.0:1584535818.928820:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535818.928834:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535818.928835:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535818.928836:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535818.928837:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535818.928839:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535818.928840:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535818.928841:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535818.928842:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584535818.976768:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584535818.976774:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584535818.976777:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584535818.976778:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584535818.976780:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584535818.976782:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584535818.976784:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584535818.976786:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535899 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535818.976820:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078419424 new x1661492078419472 00000800:00000100:0.0:1584535825.141044:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535825.141477:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535825.153523:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535818/real 1584535818] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535905 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535825.153532:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535825.153533:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535825.153534:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535825.153540:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535825.153542:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535825.153543:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535825.153544:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535825.153545:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535825.153546:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535825.153547:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535825.153548:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17624 00000100:00080000:20.0:1584535825.153551:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535825.153561:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535825.153562:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535825.153564:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535825.153565:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535825.153566:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535825.153567:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535825.153568:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535825.153569:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584535825.206358:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584535825.206362:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584535825.206364:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584535825.206365:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584535825.206366:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584535825.206368:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584535825.206370:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584535825.206371:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535905 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535825.206401:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078419472 new x1661492078419504 00000800:00020000:6.0:1584535830.916081:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584535830.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584535830.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535830.916602:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535830.928686:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535825/real 1584535825] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535912 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535830.928704:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535830.928705:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535830.928707:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535830.928713:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535830.928714:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535830.928715:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535830.928716:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535830.928717:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535830.928718:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535830.928718:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535830.928720:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17631 00000100:00080000:20.0:1584535830.928722:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535830.928732:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535830.928733:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535830.928733:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535830.928734:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535830.928735:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535830.928736:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535830.928737:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535830.928738:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584535830.968707:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584535830.968715:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584535830.968718:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584535830.968720:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584535830.968723:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584535830.968727:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584535830.968731:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584535830.968735:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535912 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535830.968783:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078419504 new x1661492078419536 00000800:00020000:0.0:1584535836.916168:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584535836.916170:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584535836.916172:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535836.916589:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535836.928640:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535830/real 1584535830] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535917 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535836.928648:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535836.928649:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535836.928651:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535836.928666:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535836.928667:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535836.928670:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535836.928671:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535836.928672:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535836.928673:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535836.928674:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535836.928677:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17636 00000100:00080000:20.0:1584535836.928678:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535836.928685:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535836.928686:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535836.928687:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535836.928688:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535836.928689:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535836.928690:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535836.928690:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535836.928692:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535836.936546:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535836.936550:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535836.936552:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535836.936553:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535836.936554:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535836.936557:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535836.936558:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535836.936560:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535917 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535836.936589:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078419536 new x1661492078419616 00000800:00020000:2.0:1584535842.916110:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535842.916112:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535842.916113:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535842.916496:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535842.928581:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535836/real 1584535836] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535923 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535842.928588:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535842.928589:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535842.928608:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535842.928614:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535842.928615:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535842.928616:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535842.928617:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535842.928618:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535842.928620:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535842.928621:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535842.928622:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17642 00000100:00080000:20.0:1584535842.928624:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535842.928633:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535842.928634:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535842.928635:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535842.928636:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535842.928637:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535842.928638:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535842.928638:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535842.928639:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535842.936559:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535842.936563:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535842.936565:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535842.936565:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535842.936567:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535842.936568:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535842.936570:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535842.936572:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535923 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535842.936601:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078419616 new x1661492078419648 00000800:00000100:0.0:1584535849.141609:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535849.142036:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535849.154093:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535842/real 1584535842] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535929 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535849.154100:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535849.154102:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535849.154103:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535849.154109:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535849.154111:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535849.154112:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535849.154114:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535849.154114:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535849.154116:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535849.154116:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535849.154118:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17648 00000100:00080000:20.0:1584535849.154120:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535849.154129:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535849.154130:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535849.154131:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535849.154132:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535849.154133:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535849.154134:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535849.154134:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535849.154135:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535849.162233:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535849.162237:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535849.162238:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535849.162239:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535849.162240:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535849.162242:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535849.162244:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535849.162246:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535929 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535849.162278:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078419648 new x1661492078419696 00000800:00020000:2.0:1584535854.916092:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535854.916098:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535854.916102:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535854.916565:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535854.928655:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535849/real 1584535849] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535936 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535854.928674:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535854.928675:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535854.928676:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535854.928681:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535854.928684:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535854.928685:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535854.928686:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535854.928686:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535854.928688:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535854.928688:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535854.928690:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17655 00000100:00080000:20.0:1584535854.928692:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535854.928700:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535854.928701:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535854.928702:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535854.928703:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535854.928703:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535854.928704:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535854.928705:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535854.928706:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535854.936891:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535854.936895:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535854.936896:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535854.936897:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535854.936898:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535854.936901:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535854.936902:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535854.936904:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535936 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584535854.936937:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078419696 new x1661492078419728 00000800:00020000:2.0:1584535860.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535860.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535860.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535860.916488:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584535860.928541:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535854/real 1584535854] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535941 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584535860.928549:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584535860.928550:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584535860.928553:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584535860.928566:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535860.928568:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535860.928571:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535860.928572:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535860.928573:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535860.928574:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584535860.928575:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584535860.928577:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17660 00000100:00080000:20.0:1584535860.928578:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584535860.928584:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535860.928585:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535860.928586:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535860.928587:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535860.928587:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535860.928589:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584535860.928589:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584535860.928590:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535860.936437:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535860.936446:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535860.936449:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535860.936451:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535860.936454:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535860.936458:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535860.936461:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535860.936465:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535941 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535860.936484:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078419728 new x1661492078419792 00000800:00020000:4.0:1584535866.916058:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535866.916071:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535866.916073:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535866.916560:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535866.928680:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535860/real 1584535860] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535947 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535866.928688:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535866.928689:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535866.928691:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535866.928697:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535866.928698:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535866.928699:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535866.928701:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535866.928702:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535866.928703:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535866.928704:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535866.928706:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17666 00000100:00080000:22.0:1584535866.928708:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535866.928718:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535866.928720:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535866.928720:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535866.928721:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535866.928722:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535866.928723:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535866.928724:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535866.928725:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535866.972445:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535866.972449:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535866.972451:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535866.972451:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535866.972453:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535866.972455:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535866.972456:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535866.972458:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535947 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535866.972492:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078419792 new x1661492078419840 00000800:00000100:0.0:1584535873.142000:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535873.142449:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535873.154555:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535866/real 1584535866] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535953 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535873.154566:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535873.154567:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535873.154569:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535873.154578:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535873.154581:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535873.154582:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535873.154584:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535873.154584:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535873.154586:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535873.154587:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535873.154589:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17672 00000100:00080000:22.0:1584535873.154592:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535873.154608:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535873.154609:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535873.154610:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535873.154611:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535873.154612:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535873.154613:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535873.154614:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535873.154615:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535873.162584:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535873.162595:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535873.162598:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535873.162601:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535873.162604:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535873.162610:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535873.162614:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535873.162618:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535953 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535873.162668:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078419840 new x1661492078419872 00000800:00020000:4.0:1584535878.916097:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535878.916104:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535878.916108:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535878.916563:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535878.928649:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535873/real 1584535873] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535960 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535878.928669:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535878.928671:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535878.928673:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535878.928681:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535878.928683:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535878.928684:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535878.928685:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535878.928686:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535878.928687:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535878.928688:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535878.928690:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17679 00000100:00080000:22.0:1584535878.928693:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535878.928704:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535878.928706:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535878.928706:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535878.928707:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535878.928708:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535878.928709:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535878.928710:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535878.928722:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535878.937547:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535878.937553:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535878.937555:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535878.937556:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535878.937558:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535878.937561:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535878.937562:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535878.937564:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535960 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535878.937614:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078419872 new x1661492078419920 00000800:00020000:2.0:1584535884.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535884.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535884.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535884.916492:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535884.928541:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535878/real 1584535878] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535965 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535884.928555:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535884.928556:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535884.928558:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535884.928574:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535884.928576:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535884.928579:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535884.928580:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535884.928581:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535884.928582:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535884.928583:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535884.928585:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17684 00000100:00080000:22.0:1584535884.928587:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535884.928594:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535884.928595:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535884.928596:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535884.928597:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535884.928598:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535884.928599:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535884.928600:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535884.928601:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535884.954764:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535884.954770:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535884.954772:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535884.954773:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535884.954774:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535884.954776:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535884.954778:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535884.954780:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535965 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535884.954809:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078419920 new x1661492078419984 00000800:00020000:4.0:1584535890.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535890.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535890.916088:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535890.916481:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535890.928571:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535884/real 1584535884] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535971 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535890.928589:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535890.928590:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535890.928591:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535890.928597:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535890.928599:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535890.928599:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535890.928600:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535890.928601:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535890.928603:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535890.928603:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535890.928605:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17690 00000100:00080000:22.0:1584535890.928607:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535890.928616:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535890.928617:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535890.928618:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535890.928619:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535890.928619:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535890.928621:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535890.928622:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535890.928635:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535890.953250:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535890.953254:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535890.953256:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535890.953257:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535890.953258:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535890.953261:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535890.953262:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535890.953264:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535971 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535890.953297:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078419984 new x1661492078420016 00000800:00000100:0.0:1584535897.141327:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535897.141818:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535897.153874:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535890/real 1584535890] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535977 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535897.153882:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535897.153884:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535897.153885:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535897.153891:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535897.153892:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535897.153893:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535897.153894:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535897.153895:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535897.153896:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535897.153897:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535897.153899:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17696 00000100:00080000:22.0:1584535897.153901:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535897.153910:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535897.153911:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535897.153912:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535897.153913:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535897.153915:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535897.153916:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535897.153917:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535897.153918:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535897.214934:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535897.214938:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535897.214939:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535897.214940:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535897.214941:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535897.214943:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535897.214945:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535897.214946:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535977 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535897.214977:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078420016 new x1661492078420064 00000800:00020000:2.0:1584535902.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535902.916081:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535902.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535902.916486:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535902.928547:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535897/real 1584535897] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535984 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535902.928555:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535902.928556:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535902.928557:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535902.928562:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535902.928564:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535902.928565:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535902.928566:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535902.928566:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535902.928568:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535902.928568:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535902.928570:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17703 00000100:00080000:22.0:1584535902.928572:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535902.928580:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535902.928581:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535902.928583:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535902.928584:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535902.928585:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535902.928586:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535902.928587:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535902.928588:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535902.952407:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535902.952411:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535902.952412:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535902.952413:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535902.952414:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535902.952416:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535902.952417:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535902.952419:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535984 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535902.952452:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078420064 new x1661492078420096 00000800:00020000:4.0:1584535908.916129:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535908.916132:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535908.916134:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535908.916568:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535908.928621:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535902/real 1584535902] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535989 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535908.928629:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535908.928630:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535908.928631:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535908.928646:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535908.928648:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535908.928655:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535908.928656:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535908.928657:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535908.928658:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535908.928659:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535908.928661:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17708 00000100:00080000:22.0:1584535908.928663:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535908.928669:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535908.928671:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535908.928671:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535908.928672:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535908.928673:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535908.928674:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535908.928675:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535908.928676:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535908.956839:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535908.956843:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535908.956845:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535908.956845:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535908.956847:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535908.956849:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535908.956851:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535908.956853:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535989 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535908.956861:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078420096 new x1661492078420176 00000800:00020000:2.0:1584535914.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535914.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535914.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535914.916489:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535914.928542:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535908/real 1584535908] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535995 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535914.928550:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535914.928551:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535914.928552:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535914.928557:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535914.928559:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535914.928559:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535914.928561:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535914.928561:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535914.928563:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535914.928563:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535914.928566:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17714 00000100:00080000:22.0:1584535914.928568:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535914.928576:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535914.928577:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535914.928578:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535914.928579:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535914.928580:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535914.928581:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535914.928582:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535914.928583:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535914.956318:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535914.956322:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535914.956323:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535914.956324:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535914.956326:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535914.956327:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535914.956329:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535914.956331:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584535995 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535914.956360:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078420176 new x1661492078420208 00000800:00000100:0.0:1584535921.141735:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535921.142195:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535921.154245:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535914/real 1584535914] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536001 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535921.154253:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535921.154255:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535921.154256:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535921.154262:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535921.154263:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535921.154264:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535921.154265:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535921.154266:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535921.154268:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535921.154269:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535921.154271:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17720 00000100:00080000:22.0:1584535921.154273:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535921.154281:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535921.154283:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535921.154283:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535921.154285:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535921.154285:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535921.154287:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535921.154287:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535921.154288:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535921.211956:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535921.211960:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535921.211961:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535921.211962:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535921.211963:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535921.211965:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535921.211967:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535921.211968:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536001 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535921.211998:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078420208 new x1661492078420240 00000800:00020000:2.0:1584535926.916092:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535926.916098:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535926.916103:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535926.916516:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535926.928663:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535921/real 1584535921] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536008 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535926.928672:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535926.928673:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535926.928674:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535926.928681:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535926.928682:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535926.928683:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535926.928685:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535926.928686:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535926.928687:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535926.928688:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535926.928690:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17727 00000100:00080000:22.0:1584535926.928692:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535926.928702:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535926.928703:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535926.928703:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535926.928705:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535926.928705:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535926.928707:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535926.928707:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535926.928709:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535926.970725:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535926.970729:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535926.970731:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535926.970732:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535926.970733:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535926.970736:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535926.970737:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535926.970739:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536008 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535926.970769:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078420240 new x1661492078420288 00000800:00020000:2.0:1584535932.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535932.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535932.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535932.916529:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535932.928584:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535926/real 1584535926] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536013 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535932.928594:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535932.928595:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535932.928597:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535932.928618:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535932.928621:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535932.928625:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535932.928626:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535932.928627:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535932.928629:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535932.928629:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535932.928632:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17732 00000100:00080000:22.0:1584535932.928634:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535932.928641:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535932.928642:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535932.928643:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535932.928644:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535932.928645:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535932.928646:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535932.928647:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535932.928648:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535932.977709:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535932.977715:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535932.977717:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535932.977719:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535932.977720:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535932.977723:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535932.977724:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535932.977726:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536013 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535932.977735:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078420288 new x1661492078420352 00000800:00020000:2.0:1584535938.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535938.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535938.916083:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535938.916559:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535938.928641:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535932/real 1584535932] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536019 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535938.928651:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535938.928653:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535938.928656:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535938.928663:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535938.928665:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535938.928666:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535938.928668:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535938.928668:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535938.928670:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535938.928671:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535938.928673:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17738 00000100:00080000:22.0:1584535938.928676:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535938.928690:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535938.928691:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535938.928692:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535938.928693:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535938.928694:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535938.928695:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535938.928696:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535938.928697:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535938.974715:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535938.974722:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535938.974724:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535938.974725:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535938.974727:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535938.974729:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535938.974731:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535938.974733:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536019 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535938.974763:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078420352 new x1661492078420400 00000800:00000100:0.0:1584535945.140958:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584535945.141413:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535945.153470:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535938/real 1584535938] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536025 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535945.153486:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535945.153488:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535945.153490:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535945.153499:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535945.153501:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535945.153502:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535945.153503:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535945.153504:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535945.153506:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535945.153507:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535945.153509:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17744 00000100:00080000:22.0:1584535945.153513:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535945.153529:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535945.153530:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535945.153531:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535945.153533:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535945.153533:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535945.153534:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535945.153535:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535945.153536:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535945.200717:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535945.200724:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535945.200727:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535945.200728:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535945.200730:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535945.200733:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535945.200734:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535945.200736:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536025 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535945.200766:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078420400 new x1661492078420432 00000800:00020000:2.0:1584535950.916054:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535950.916057:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535950.916059:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535950.916473:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535950.928528:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535945/real 1584535945] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536032 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535950.972403:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535950.972404:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535950.972405:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535950.972410:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535950.972411:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535950.972412:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535950.972413:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535950.972414:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535950.972415:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535950.972416:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535950.972418:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17751 00000100:00080000:22.0:1584535950.972420:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535950.972430:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535950.972431:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535950.972432:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535950.972433:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535950.972434:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535950.972435:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535950.972437:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535950.972438:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535951.024659:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535951.024663:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535951.024664:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535951.024665:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535951.024666:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535951.024668:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535951.024670:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535951.024672:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536032 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535951.024701:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078420432 new x1661492078420464 00000800:00020000:6.0:1584535956.916072:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584535956.916075:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584535956.916077:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535956.916463:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535956.928521:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535951/real 1584535951] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536038 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535956.928531:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535956.928532:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535956.928534:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535956.928553:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535956.928555:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535956.928563:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535956.928564:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535956.928564:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535956.928566:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535956.928567:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535956.928569:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17756 00000100:00080000:22.0:1584535956.928571:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535956.928579:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535956.928580:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535956.928581:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535956.928582:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535956.928584:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535956.928585:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535956.928586:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535956.928587:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535956.969914:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535956.969920:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535956.969922:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535956.969923:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535956.969924:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535956.969926:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535956.969928:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535956.969930:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536038 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535956.969938:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078420464 new x1661492078420544 00000800:00020000:2.0:1584535962.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535962.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535962.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535962.916479:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535962.928531:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535956/real 1584535956] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536043 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535962.928539:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535962.928540:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535962.928541:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535962.928546:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535962.928548:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535962.928549:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535962.928550:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535962.928550:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535962.928552:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535962.928552:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535962.928554:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17762 00000100:00080000:22.0:1584535962.928556:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535962.928565:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535962.928566:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535962.928568:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535962.928569:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535962.928570:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535962.928571:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535962.928572:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535962.928573:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535962.975095:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535962.975098:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535962.975100:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535962.975101:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535962.975102:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535962.975104:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535962.975106:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535962.975108:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536043 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535962.975116:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078420544 new x1661492078420576 00000800:00020000:4.0:1584535968.916059:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535968.916062:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535968.916064:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535968.916472:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535968.928525:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535962/real 1584535962] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536049 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535968.928533:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535968.928535:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535968.928536:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535968.928541:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535968.928543:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535968.928543:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535968.928545:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535968.928546:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535968.928547:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535968.928548:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535968.928549:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17768 00000100:00080000:22.0:1584535968.928551:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535968.928561:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535968.928562:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535968.928563:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535968.928564:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535968.928565:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535968.928566:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535968.928567:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535968.928568:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535968.975301:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535968.975305:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535968.975307:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535968.975308:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535968.975309:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535968.975311:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535968.975312:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535968.975314:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536049 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535968.975322:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078420576 new x1661492078420624 00000800:00020000:2.0:1584535974.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535974.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535974.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535974.916495:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535974.928557:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535968/real 1584535968] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536055 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535974.928565:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535974.928566:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535974.928567:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535974.928572:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535974.928574:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535974.928575:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535974.928576:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535974.928577:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535974.928578:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535974.928578:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535974.928581:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17774 00000100:00080000:22.0:1584535974.928583:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535974.928592:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535974.928593:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535974.928594:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535974.928595:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535974.928595:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535974.928597:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535974.928597:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535974.928598:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535974.983297:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535974.983301:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535974.983302:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535974.983303:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535974.983304:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535974.983306:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535974.983308:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535974.983310:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536055 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535974.983339:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078420624 new x1661492078420656 00000800:00020000:2.0:1584535980.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535980.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535980.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535980.916523:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535980.928578:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535974/real 1584535974] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536061 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535980.928586:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535980.928587:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535980.928588:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535980.928603:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535980.928605:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535980.928608:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535980.928609:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535980.928610:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535980.928613:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535980.928614:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535980.928615:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17780 00000100:00080000:22.0:1584535980.928618:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535980.928623:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535980.928624:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535980.928625:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535980.928626:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535980.928627:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535980.928628:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535980.928628:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535980.928629:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535980.973849:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535980.973853:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535980.973854:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535980.973855:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535980.973856:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535980.973859:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535980.973861:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535980.973862:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536061 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535980.973892:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078420656 new x1661492078420720 00000800:00020000:4.0:1584535986.916062:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584535986.916066:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584535986.916068:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535986.916504:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535986.928567:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535980/real 1584535980] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536067 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535986.928577:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535986.928579:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535986.928581:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535986.928589:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535986.928591:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535986.928591:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535986.928594:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535986.928594:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535986.928596:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535986.928597:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535986.928599:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17786 00000100:00080000:22.0:1584535986.928601:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535986.928615:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535986.928616:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535986.928617:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535986.928618:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535986.928619:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535986.928620:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535986.928621:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535986.928622:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535986.971848:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535986.971854:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535986.971857:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535986.971858:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535986.971859:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535986.971862:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535986.971863:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535986.971865:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536067 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535986.971895:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078420720 new x1661492078420768 00000800:00020000:2.0:1584535992.916098:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535992.916104:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535992.916109:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535992.916639:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535992.928736:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535986/real 1584535986] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536073 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535992.928757:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535992.928759:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535992.928761:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535992.928769:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535992.928772:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535992.928773:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535992.928774:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535992.928775:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535992.928777:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535992.928778:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535992.928780:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17792 00000100:00080000:22.0:1584535992.928783:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535992.928798:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535992.928800:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535992.928801:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535992.928802:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535992.928803:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535992.928804:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535992.928805:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535992.928806:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535992.989285:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535992.989291:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535992.989294:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535992.989295:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535992.989297:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535992.989299:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535992.989301:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535992.989303:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536073 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535992.989332:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078420768 new x1661492078420800 00000800:00020000:2.0:1584535998.916073:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584535998.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584535998.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584535998.916481:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584535998.928576:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535992/real 1584535992] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536079 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584535998.928598:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584535998.928599:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584535998.928602:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584535998.928609:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535998.928611:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535998.928612:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535998.928614:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535998.928614:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535998.928616:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584535998.928617:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584535998.928619:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17798 00000100:00080000:22.0:1584535998.928622:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584535998.928636:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535998.928637:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535998.928638:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535998.928639:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535998.928640:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535998.928641:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584535998.928642:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584535998.928643:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584535998.981895:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584535998.981901:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584535998.981904:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584535998.981905:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584535998.981907:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584535998.981910:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584535998.981911:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584535998.981913:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536079 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584535998.981943:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078420800 new x1661492078420848 00000800:00020000:2.0:1584536004.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536004.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536004.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536004.916498:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584536004.928554:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584535998/real 1584535998] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536085 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584536004.928567:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584536004.928569:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584536004.928570:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584536004.928588:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536004.928590:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536004.928593:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536004.928594:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536004.928595:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536004.928597:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584536004.928598:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584536004.928600:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17804 00000100:00080000:22.0:1584536004.928602:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584536004.928609:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536004.928610:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536004.928611:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536004.928612:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536004.928613:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536004.928614:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536004.928615:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536004.928616:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536004.966719:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536004.966725:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536004.966727:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536004.966728:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536004.966729:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536004.966732:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536004.966733:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536004.966735:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536085 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584536004.966743:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078420848 new x1661492078420912 00000800:00020000:2.0:1584536010.916073:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536010.916075:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536010.916076:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536010.916526:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584536010.928637:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536004/real 1584536004] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536091 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584536010.928646:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584536010.928647:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584536010.928648:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584536010.928654:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536010.928656:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536010.928657:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536010.928658:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536010.928659:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536010.928660:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584536010.928661:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584536010.928662:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17810 00000100:00080000:6.0:1584536010.928664:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584536010.928682:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536010.928683:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536010.928684:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536010.928685:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536010.928686:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536010.928688:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536010.928690:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536010.928692:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536010.973429:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536010.973433:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536010.973435:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536010.973436:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536010.973437:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536010.973439:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536010.973440:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536010.973442:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536091 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584536010.973451:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078420912 new x1661492078420944 00000800:00020000:2.0:1584536016.916081:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536016.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536016.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536016.916577:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584536016.928655:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536010/real 1584536010] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536097 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584536016.928663:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584536016.928664:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584536016.928665:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584536016.928671:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536016.928673:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536016.928674:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536016.928675:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536016.928676:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536016.928677:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584536016.928677:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584536016.928679:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17816 00000100:00080000:6.0:1584536016.928681:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584536016.928690:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536016.928691:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536016.928692:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536016.928694:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536016.928695:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536016.928696:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536016.928696:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536016.928697:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536016.970524:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536016.970528:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536016.970529:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536016.970530:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536016.970531:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536016.970534:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536016.970535:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536016.970537:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536097 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584536016.970547:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078420944 new x1661492078420992 00000800:00020000:6.0:1584536022.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584536022.916081:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584536022.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536022.916545:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584536022.928629:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536016/real 1584536016] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536103 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584536022.928647:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584536022.954086:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584536022.954088:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584536022.954091:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536022.954093:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536022.954093:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536022.954094:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536022.954095:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536022.954096:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584536022.954097:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584536022.954098:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17822 00000100:00080000:6.0:1584536022.954101:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584536022.954108:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536022.954109:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536022.954110:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536022.954111:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536022.954111:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536022.954112:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536022.954113:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536022.954114:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536023.007679:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536023.007683:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536023.007684:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536023.007685:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536023.007686:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536023.007688:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536023.007689:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536023.025604:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536103 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584536023.025625:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078420992 new x1661492078421024 00000800:00020000:2.0:1584536028.916064:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536028.916067:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536028.916069:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536028.916451:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584536028.928527:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536023/real 1584536023] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536110 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584536028.928535:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584536028.928536:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584536028.928537:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584536028.928553:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536028.928554:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536028.928557:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536028.928558:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536028.928559:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536028.928560:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584536028.928561:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584536028.928563:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17828 00000100:00080000:8.0:1584536028.928565:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584536028.928570:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536028.928572:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536028.928572:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536028.928574:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536028.928575:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536028.928576:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536028.928577:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536028.928578:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536028.976886:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536028.976890:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536028.976891:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536028.976892:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536028.976893:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536028.976895:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536028.976897:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536028.976898:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536110 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584536028.976928:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078421024 new x1661492078421104 00000800:00020000:2.0:1584536034.916093:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536034.916095:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536034.916097:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536034.916504:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584536034.928589:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536028/real 1584536028] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536115 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584536034.928608:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584536034.928609:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584536034.928610:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584536034.928616:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536034.928618:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536034.928619:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536034.928620:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536034.928620:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536034.928622:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584536034.928622:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584536034.928624:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17834 00000100:00080000:8.0:1584536034.928626:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584536034.928635:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536034.928637:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536034.928638:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536034.928639:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536034.928639:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536034.928640:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536034.928641:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536034.928642:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536034.965849:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536034.965853:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536034.965854:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536034.965855:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536034.965857:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536034.965859:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536034.965861:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536034.965863:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536115 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584536034.965891:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078421104 new x1661492078421136 00000800:00020000:2.0:1584536040.916121:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536040.916124:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536040.916126:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536040.916640:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584536040.928688:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536034/real 1584536034] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536121 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584536040.928696:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584536040.928697:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584536040.928698:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584536040.928703:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536040.928704:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536040.928705:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536040.928706:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536040.928707:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536040.928708:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584536040.928709:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584536040.928710:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17840 00000100:00080000:8.0:1584536040.928713:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584536040.928722:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536040.928723:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536040.928724:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536040.928725:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536040.928726:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536040.928727:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536040.928727:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536040.928729:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536040.978889:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536040.978893:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536040.978895:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536040.978895:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536040.978897:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536040.978898:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536040.978900:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536040.978902:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536121 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584536040.978930:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078421136 new x1661492078421168 00000800:00020000:2.0:1584536046.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536046.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536046.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536046.916472:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584536046.928562:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536040/real 1584536040] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536127 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584536046.928582:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584536046.928583:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584536046.928584:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584536046.928590:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536046.928592:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536046.928593:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536046.928594:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536046.928594:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536046.928596:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584536046.928597:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584536046.928599:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17846 00000100:00080000:8.0:1584536046.928601:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584536046.928610:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536046.928612:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536046.928612:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536046.928614:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536046.928614:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536046.928616:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536046.928616:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536046.928617:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536046.973069:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536046.973074:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536046.973076:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536046.973077:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536046.973078:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536046.973080:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536046.973082:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536046.973084:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536127 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584536046.973113:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078421168 new x1661492078421216 00000800:00020000:2.0:1584536052.916061:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536052.916064:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536052.916066:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536052.916446:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584536052.928538:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536046/real 1584536046] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536133 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584536052.928547:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584536052.928548:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584536052.928549:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584536052.928565:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536052.928567:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536052.928571:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536052.928572:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536052.928574:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536052.928575:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584536052.928576:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584536052.928578:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17852 00000100:00080000:8.0:1584536052.928580:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584536052.928586:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536052.928587:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536052.928588:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536052.928589:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536052.928590:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536052.928591:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536052.928592:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536052.928593:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536052.974707:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536052.974711:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536052.974713:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536052.974714:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536052.974715:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536052.974718:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536052.974720:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536052.974721:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536133 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584536052.974750:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078421216 new x1661492078421280 00000800:00020000:4.0:1584536058.916119:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536058.916122:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536058.916124:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536058.916598:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584536058.928739:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536052/real 1584536052] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536139 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584536058.928750:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584536058.928752:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584536058.928754:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584536058.928761:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536058.928763:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536058.928765:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536058.928766:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536058.928767:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536058.928769:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584536058.928769:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584536058.928771:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17858 00000100:00080000:8.0:1584536058.928775:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584536058.928789:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536058.928790:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536058.928791:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536058.928793:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536058.928793:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536058.928794:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536058.928795:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536058.928796:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536058.983073:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536058.983079:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536058.983082:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536058.983083:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536058.983084:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536058.983087:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536058.983088:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536058.983090:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536139 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584536058.983124:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078421280 new x1661492078421328 00000800:00020000:6.0:1584536064.916055:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584536064.916058:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584536064.916060:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536064.916496:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584536064.928570:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536058/real 1584536058] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536145 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584536064.928579:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584536064.928581:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584536064.928582:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584536064.928591:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536064.928592:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536064.928593:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536064.928595:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536064.928595:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536064.928597:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584536064.928598:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584536064.928600:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17864 00000100:00080000:8.0:1584536064.928602:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584536064.928616:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536064.928618:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536064.928618:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536064.928620:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536064.928620:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536064.928621:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536064.928622:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536064.928623:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536064.973563:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536064.973569:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536064.973571:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536064.973572:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536064.973573:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536064.973576:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536064.973577:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536064.973579:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536145 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584536064.973609:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078421328 new x1661492078421360 00000800:00000100:0.0:1584536071.141097:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536071.141523:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584536071.153597:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536064/real 1584536064] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536151 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584536071.153605:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584536071.153607:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584536071.153609:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584536071.153615:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536071.153617:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536071.153617:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536071.153619:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536071.153619:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536071.153621:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584536071.153621:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584536071.153623:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17870 00000100:00080000:8.0:1584536071.153625:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584536071.153637:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536071.153638:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536071.153638:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536071.153640:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536071.153640:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536071.153641:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536071.153642:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536071.153643:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536071.201865:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536071.201871:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536071.201873:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536071.201874:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536071.201875:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536071.201879:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536071.201881:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536071.201882:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536151 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584536071.201916:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078421360 new x1661492078421392 00000800:00020000:2.0:1584536076.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536076.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536076.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536076.916524:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584536076.928602:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536071/real 1584536071] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536158 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584536076.928611:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584536076.928612:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584536076.928613:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584536076.928628:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536076.928630:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536076.928633:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536076.928634:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536076.928634:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536076.928636:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584536076.928636:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584536076.928638:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17877 00000100:00080000:8.0:1584536076.928640:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584536076.928646:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536076.928648:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536076.928648:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536076.928649:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536076.928650:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536076.928651:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536076.928652:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536076.928653:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536076.973662:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536076.973667:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536076.973669:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536076.973670:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536076.973671:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536076.973673:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536076.973675:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536076.973676:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536158 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584536076.973708:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078421392 new x1661492078421472 00000800:00020000:2.0:1584536082.916100:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536082.916107:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536082.916112:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536082.916537:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584536082.928628:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536076/real 1584536076] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536163 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584536082.928646:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584536082.928647:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584536082.928649:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584536082.928654:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536082.928656:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536082.928657:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536082.928658:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536082.928658:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536082.928660:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584536082.928660:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584536082.928662:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17882 00000100:00080000:8.0:1584536082.928664:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584536082.928672:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536082.928673:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536082.928673:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536082.928674:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536082.928675:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536082.928676:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536082.928678:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536082.928679:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536082.976883:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536082.976887:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536082.976889:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536082.976889:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536082.976891:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536082.976892:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536082.976894:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536082.976896:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536163 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584536082.976925:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078421472 new x1661492078421504 00000800:00020000:4.0:1584536088.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536088.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536088.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536088.916545:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584536088.928621:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536082/real 1584536082] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536169 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584536088.928628:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584536088.928630:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584536088.928631:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584536088.928637:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536088.928638:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536088.928639:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536088.928641:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536088.928641:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536088.928642:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584536088.928643:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584536088.928645:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17888 00000100:00080000:8.0:1584536088.928647:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584536088.928655:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536088.928657:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536088.928657:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536088.928659:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536088.928660:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536088.928661:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536088.928662:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536088.928663:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536088.984010:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536088.984014:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536088.984016:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536088.984017:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536088.984018:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536088.984021:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536088.984022:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536088.984024:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536169 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584536088.984070:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078421504 new x1661492078421552 00000800:00000100:0.0:1584536095.141615:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536095.142044:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584536095.154114:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536088/real 1584536088] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536175 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584536095.154122:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584536095.154123:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584536095.154124:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584536095.154129:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536095.154131:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536095.154131:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536095.154132:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536095.154133:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536095.154134:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584536095.154135:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584536095.154137:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17894 00000100:00080000:8.0:1584536095.154138:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584536095.154146:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536095.154147:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536095.154149:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536095.154150:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536095.154151:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536095.154152:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536095.154152:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536095.154153:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536095.210470:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536095.210478:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536095.210481:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536095.210483:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536095.210486:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536095.210490:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536095.210494:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536095.210498:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536175 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584536095.210545:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078421552 new x1661492078421584 00000800:00020000:2.0:1584536100.916069:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536100.916072:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536100.916074:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536100.916515:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584536100.928591:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536095/real 1584536095] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536182 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584536100.928598:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584536100.928599:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584536100.928601:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584536100.928615:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536100.928616:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536100.928620:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536100.928621:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536100.928622:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536100.928623:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584536100.928624:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584536100.928625:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17901 00000100:00080000:8.0:1584536100.928628:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584536100.928634:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536100.928635:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536100.928636:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536100.928637:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536100.928637:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536100.928639:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536100.928639:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536100.928640:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536100.979725:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536100.979729:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536100.979730:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536100.979731:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536100.979732:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536100.979734:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536100.979735:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536100.979737:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536182 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584536100.979766:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078421584 new x1661492078421648 00000800:00020000:6.0:1584536106.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584536106.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584536106.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536106.916473:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584536106.928569:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536100/real 1584536100] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536187 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584536106.928590:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584536106.928592:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584536106.928593:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584536106.928601:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536106.928603:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536106.928604:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536106.928605:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536106.928606:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536106.928607:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584536106.928608:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584536106.928611:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17906 00000100:00080000:8.0:1584536106.928614:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584536106.928628:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536106.928630:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536106.928630:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536106.928632:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536106.928632:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536106.928633:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536106.928634:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536106.928635:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536106.985054:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536106.985060:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536106.985062:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536106.985063:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536106.985065:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536106.985069:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536106.985070:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536106.985072:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536187 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584536106.985082:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078421648 new x1661492078421696 00000800:00020000:2.0:1584536112.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536112.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536112.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536112.916540:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584536112.928597:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536106/real 1584536106] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536193 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584536112.928608:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584536112.928609:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584536112.928611:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584536112.928619:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536112.928621:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536112.928621:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536112.928623:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536112.928624:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536112.928626:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584536112.928627:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584536112.928629:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17912 00000100:00080000:8.0:1584536112.928632:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584536112.928648:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536112.928650:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536112.928651:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536112.928652:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536112.928652:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536112.928654:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536112.928654:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536112.928656:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536112.982269:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536112.982275:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536112.982278:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536112.982279:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536112.982281:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536112.982283:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536112.982285:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536112.982287:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536193 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584536112.982317:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078421696 new x1661492078421728 00000800:00000100:0.0:1584536119.141317:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536119.141781:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584536119.153859:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536112/real 1584536112] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536199 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584536119.153868:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584536119.153870:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584536119.153872:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584536119.153879:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536119.153880:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536119.153881:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536119.153883:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536119.153884:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536119.153886:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584536119.153887:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584536119.153889:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17918 00000100:00080000:8.0:1584536119.153892:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584536119.153903:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536119.153904:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536119.153905:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536119.153906:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536119.153907:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536119.153908:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536119.153909:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536119.153921:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536119.198584:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536119.198591:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536119.198593:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536119.198594:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536119.198597:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536119.198599:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536119.198601:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536119.198603:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536199 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584536119.198632:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078421728 new x1661492078421776 00000800:00020000:4.0:1584536124.916061:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536124.916064:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536124.916067:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536124.916926:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584536124.928989:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536119/real 1584536119] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536206 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584536124.928997:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584536124.928998:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584536124.928999:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584536124.929013:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536124.929016:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536124.929020:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536124.929021:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536124.929022:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536124.929023:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584536124.929024:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584536124.929026:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17925 00000100:00080000:8.0:1584536124.929045:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584536124.929051:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536124.929052:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536124.929052:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536124.929054:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536124.929055:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536124.929056:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536124.929056:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536124.929057:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536124.976739:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536124.976743:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536124.976745:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536124.976745:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536124.976747:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536124.976750:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536124.976751:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536124.976753:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536206 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584536124.976782:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078421776 new x1661492078421840 00000800:00020000:6.0:1584536130.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584536130.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584536130.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536130.916487:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584536130.928563:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536124/real 1584536124] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536211 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584536130.928571:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584536130.928572:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584536130.928575:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584536130.928580:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536130.928582:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536130.928583:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536130.928584:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536130.928584:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536130.928586:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584536130.928586:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584536130.928588:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17930 00000100:00080000:8.0:1584536130.928590:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584536130.928599:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536130.928600:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536130.928600:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536130.928602:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536130.928602:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536130.928603:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536130.928604:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536130.928605:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536130.983745:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536130.983749:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536130.983750:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536130.983751:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536130.983752:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536130.983754:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536130.983756:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536130.983757:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536211 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584536130.983790:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078421840 new x1661492078421872 00000800:00020000:2.0:1584536136.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536136.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536136.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536136.916507:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584536136.928558:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536130/real 1584536130] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536217 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584536136.928571:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584536136.928572:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584536136.928573:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584536136.928578:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536136.928580:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536136.928581:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536136.928582:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536136.928582:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536136.928584:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584536136.928584:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584536136.928586:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17936 00000100:00080000:8.0:1584536136.928588:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584536136.928597:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536136.928598:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536136.928598:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536136.928600:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536136.928600:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536136.928602:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536136.928602:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536136.928603:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536136.982187:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536136.982191:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536136.982192:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536136.982193:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536136.982194:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536136.982196:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536136.982198:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536136.982199:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536217 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584536136.982232:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078421872 new x1661492078421920 00000800:00000100:2.0:1584536143.141329:0:4532:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536143.141936:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584536143.153988:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536136/real 1584536136] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536223 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584536143.153996:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584536143.153997:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584536143.153998:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584536143.154004:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536143.154005:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536143.154006:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536143.154007:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536143.154008:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536143.154009:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584536143.154010:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584536143.154012:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17942 00000100:00080000:8.0:1584536143.154014:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584536143.154022:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536143.154023:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536143.154024:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536143.154025:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536143.154025:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536143.154074:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536143.154076:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536143.154077:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536143.229481:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536143.229485:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536143.229487:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536143.229488:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536143.229489:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536143.229492:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536143.229493:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536143.229495:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536223 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584536143.229524:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078421920 new x1661492078421952 00000800:00020000:6.0:1584536148.916202:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584536148.916205:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584536148.916206:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536148.916614:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584536148.928667:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536143/real 1584536143] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536230 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584536148.928675:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584536148.928676:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584536148.928677:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584536148.928691:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536148.928692:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536148.928702:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536148.928704:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536148.928704:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536148.928706:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584536148.928706:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584536148.928708:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17949 00000100:00080000:8.0:1584536148.928710:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584536148.928715:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536148.928716:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536148.928717:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536148.928718:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536148.928720:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536148.928721:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536148.928721:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584536148.928723:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584536148.936362:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584536148.936366:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584536148.936368:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584536148.936369:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584536148.936371:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584536148.936373:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584536148.936374:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584536148.936376:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536230 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584536148.936384:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078421952 new x1661492078422032 00000800:00020000:8.0:1584536154.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584536154.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584536154.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536154.916495:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584536154.928550:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536148/real 1584536148] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536235 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584536154.928558:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584536154.928559:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584536154.928560:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584536154.928566:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536154.928568:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536154.928568:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536154.928570:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536154.928570:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536154.928572:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584536154.928572:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584536154.928574:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17954 00000100:00080000:10.0:1584536154.928577:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584536154.928586:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536154.928588:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536154.928588:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536154.928590:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536154.928590:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536154.928591:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536154.928592:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536154.928593:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584536154.936310:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584536154.936314:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584536154.936316:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584536154.936316:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584536154.936318:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584536154.936320:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584536154.936321:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584536154.936323:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536235 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584536154.936351:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078422032 new x1661492078422064 00000800:00020000:4.0:1584536160.916053:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536160.916056:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536160.916058:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536160.916646:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584536160.928698:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536154/real 1584536154] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536241 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584536160.928705:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584536160.928706:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584536160.928707:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584536160.928713:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536160.928714:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536160.928715:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536160.928716:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536160.928717:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536160.928718:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584536160.928719:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584536160.928721:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17960 00000100:00080000:10.0:1584536160.928723:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584536160.928731:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536160.928732:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536160.928733:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536160.928734:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536160.928735:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536160.928736:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536160.928736:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536160.928738:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584536160.936290:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584536160.936293:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584536160.936295:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584536160.936296:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584536160.936297:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584536160.936299:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584536160.936300:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584536160.936302:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536241 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584536160.936330:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078422064 new x1661492078422096 00000800:00000100:0.0:1584536167.141929:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536167.142381:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584536167.154460:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536160/real 1584536160] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536247 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584536167.154469:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584536167.154470:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584536167.154472:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584536167.154479:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536167.154481:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536167.154481:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536167.154483:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536167.154484:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536167.154486:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584536167.154487:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584536167.154489:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17966 00000100:00080000:10.0:1584536167.154491:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584536167.154502:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536167.154504:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536167.154504:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536167.154505:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536167.154506:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536167.154507:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536167.154508:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536167.154509:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584536167.162359:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584536167.162365:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584536167.162367:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584536167.162368:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584536167.162370:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584536167.162372:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584536167.162373:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584536167.162375:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536247 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584536167.162404:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078422096 new x1661492078422144 00000800:00020000:6.0:1584536172.916064:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584536172.938880:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584536172.963063:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536172.963520:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584536172.975599:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536167/real 1584536167] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536254 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584536172.975609:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584536172.975610:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584536172.975612:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584536172.975631:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536172.975633:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536172.975638:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536172.975639:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536172.975640:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536172.975642:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584536172.975642:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584536172.975644:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17973 00000100:00080000:10.0:1584536172.975647:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584536172.975654:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536172.975655:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536172.975656:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536172.975657:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536172.975658:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536172.975660:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536172.975660:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536172.975661:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584536173.003467:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584536173.003474:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584536173.003477:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584536173.003478:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584536173.003479:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584536173.003482:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584536173.003483:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584536173.003486:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536254 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584536173.003494:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078422144 new x1661492078422208 00000800:00020000:4.0:1584536178.916058:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536178.916061:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536178.916063:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536178.916465:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584536178.928520:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536173/real 1584536173] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536260 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584536178.928529:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584536178.928531:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584536178.928532:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584536178.928539:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536178.928541:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536178.928542:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536178.928543:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536178.928544:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536178.928546:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584536178.928547:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584536178.928549:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17978 00000100:00080000:10.0:1584536178.928551:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584536178.928565:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536178.928566:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536178.928567:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536178.928568:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536178.928569:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536178.928570:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536178.928571:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536178.928573:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584536178.953359:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584536178.953366:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584536178.953368:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584536178.953369:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584536178.953371:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584536178.953373:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584536178.953375:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584536178.953377:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536260 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584536178.953386:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078422208 new x1661492078422256 00000800:00020000:2.0:1584536184.916052:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536184.916054:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536184.916056:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536184.916463:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584536184.928519:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536178/real 1584536178] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536265 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584536184.928527:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584536184.928529:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584536184.928530:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584536184.928536:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536184.928538:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536184.928539:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536184.928540:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536184.928541:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536184.928543:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584536184.928543:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584536184.928545:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17984 00000100:00080000:10.0:1584536184.928547:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584536184.928560:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536184.928561:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536184.928562:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536184.928563:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536184.928563:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536184.928565:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536184.928565:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536184.928566:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584536184.937045:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584536184.937050:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584536184.937052:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584536184.937053:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584536184.937055:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584536184.937057:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584536184.937059:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584536184.937061:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536265 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584536184.937070:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078422256 new x1661492078422288 00000800:00000100:0.0:1584536191.141113:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536191.141532:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584536191.153582:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536184/real 1584536184] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536271 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584536191.153591:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584536191.153592:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584536191.153593:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584536191.153599:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536191.153600:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536191.153601:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536191.153602:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536191.153603:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536191.153604:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584536191.153605:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584536191.153607:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17990 00000100:00080000:10.0:1584536191.153608:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584536191.153618:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536191.153619:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536191.153620:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536191.153621:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536191.153621:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536191.153623:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536191.153623:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536191.153624:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584536191.162439:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584536191.162443:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584536191.162445:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584536191.162446:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584536191.162447:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584536191.162449:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584536191.162450:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584536191.162452:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536271 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584536191.162481:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078422288 new x1661492078422320 00000800:00020000:2.0:1584536196.916045:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536196.916047:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536196.916049:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536196.916461:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584536196.928512:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536191/real 1584536191] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584536196.928520:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584536196.928521:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584536196.928523:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584536196.928538:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536196.928539:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536196.928542:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536196.928543:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536196.928544:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536196.928545:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584536196.928546:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584536196.928547:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17997 00000100:00080000:10.0:1584536196.928550:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584536196.928555:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536196.928556:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536196.928557:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536196.928558:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536196.928559:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536196.928560:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536196.928561:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536196.928563:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584536196.950501:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584536196.950505:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584536196.950507:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584536196.950508:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584536196.950509:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584536196.950511:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584536196.950512:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584536196.950514:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584536196.950543:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078422320 new x1661492078422400 00000800:00020000:4.0:1584536202.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536202.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536202.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536202.916460:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584536202.928515:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536196/real 1584536196] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584536202.928522:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584536202.928523:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584536202.928525:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584536202.928530:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536202.928532:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536202.928533:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536202.928534:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536202.928535:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536202.928536:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584536202.928537:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584536202.928538:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18002 00000100:00080000:10.0:1584536202.928540:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584536202.928549:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536202.928550:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536202.928550:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536202.928552:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536202.928553:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536202.928554:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536202.928555:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536202.928556:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584536202.937112:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584536202.937116:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584536202.937117:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584536202.937118:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584536202.937120:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584536202.937122:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584536202.937124:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584536202.937126:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584536202.937135:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078422400 new x1661492078422432 00000800:00020000:6.0:1584536208.916204:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584536208.916207:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584536208.916209:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536208.916613:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584536208.928690:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536202/real 1584536202] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584536208.928698:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584536208.928699:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584536208.928701:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584536208.928707:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584536208.928709:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584536208.928710:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584536208.928711:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584536208.928712:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584536208.928714:0:4532:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584536208.928714:0:4532:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584536208.928716:0:4532:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18008 00000100:00080000:2.0:1584536208.928718:0:4532:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584536208.928728:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584536208.928729:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584536208.928730:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584536208.928731:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584536208.928732:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584536208.928733:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584536208.928733:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584536208.928735:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584536208.937214:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584536208.937218:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584536208.937220:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584536208.937221:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584536208.937222:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584536208.937224:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584536208.937225:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584536208.937227:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584536208.937236:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078422432 new x1661492078422480 00000800:00000100:0.0:1584536215.141629:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536215.142069:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584536215.154137:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536208/real 1584536208] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584536215.154145:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584536215.154146:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584536215.154147:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584536215.154154:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584536215.154155:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584536215.154157:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584536215.154158:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584536215.154159:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584536215.154160:0:4532:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584536215.154161:0:4532:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584536215.154162:0:4532:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18014 00000100:00080000:2.0:1584536215.154164:0:4532:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584536215.154173:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584536215.154174:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584536215.154175:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584536215.154176:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584536215.154176:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584536215.154178:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584536215.154178:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584536215.154179:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584536215.198075:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584536215.198079:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584536215.198080:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584536215.198081:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584536215.198082:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584536215.198084:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584536215.198086:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584536215.198087:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584536215.198096:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078422480 new x1661492078422512 00000800:00020000:2.0:1584536220.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536220.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536220.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536220.916584:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584536220.928630:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536215/real 1584536215] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584536220.928638:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584536220.928644:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584536220.928645:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584536220.928662:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536220.928663:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536220.928666:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536220.928668:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536220.928668:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536220.928669:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584536220.928670:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584536220.928672:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18021 00000100:00080000:4.0:1584536220.928674:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584536220.928679:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536220.928680:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536220.928681:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536220.928682:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536220.928683:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536220.928685:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536220.928690:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536220.928691:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584536220.953316:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584536220.953320:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584536220.953322:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584536220.953322:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584536220.953325:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584536220.953326:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584536220.953328:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584536220.953330:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584536220.953359:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078422512 new x1661492078422576 00000800:00020000:6.0:1584536226.916091:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584536226.916094:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584536226.916097:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536226.916572:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584536226.928665:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536220/real 1584536220] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584536226.928686:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584536226.928687:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584536226.928689:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584536226.928697:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536226.928699:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536226.928699:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536226.928701:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536226.928702:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536226.928703:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584536226.928704:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584536226.928706:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18026 00000100:00080000:4.0:1584536226.928709:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584536226.928722:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536226.928723:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536226.928724:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536226.928726:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536226.928727:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536226.928728:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536226.928728:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536226.928730:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536226.949008:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536226.949015:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536226.949017:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536226.949018:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536226.949020:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536226.949022:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536226.949024:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536226.949026:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584536226.949068:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078422576 new x1661492078422624 00000800:00020000:2.0:1584536232.916072:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536232.916075:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536232.916077:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536232.916470:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584536232.928526:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536226/real 1584536226] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584536232.928536:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584536232.928537:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584536232.928539:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584536232.928546:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536232.928548:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536232.928549:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536232.928550:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536232.928551:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536232.928553:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584536232.928554:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584536232.928556:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18032 00000100:00080000:4.0:1584536232.928558:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584536232.928573:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536232.928575:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536232.928576:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536232.928577:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536232.928578:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536232.928579:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536232.928580:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536232.928581:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536232.975100:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536232.975107:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536232.975109:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536232.975110:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536232.975113:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536232.975115:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536232.975117:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536232.975119:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584536232.975127:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078422624 new x1661492078422656 00000800:00000100:0.0:1584536239.140981:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536239.141394:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584536239.153445:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536232/real 1584536232] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536319 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584536239.153453:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584536239.153454:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584536239.153455:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584536239.153460:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536239.153462:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536239.153463:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536239.153464:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536239.153464:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536239.153466:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584536239.153466:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584536239.153468:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18038 00000100:00080000:4.0:1584536239.153471:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584536239.153480:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536239.153481:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536239.153482:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536239.153483:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536239.153484:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536239.153485:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536239.153486:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536239.153487:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536239.180720:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536239.180726:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536239.180728:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536239.180729:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536239.180731:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536239.180733:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536239.180735:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536239.180736:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536319 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584536239.180747:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078422656 new x1661492078422704 00000800:00020000:2.0:1584536244.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536244.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536244.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536244.916516:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584536244.928595:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536239/real 1584536239] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536326 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584536244.928603:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584536244.928604:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584536244.928605:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584536244.928620:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536244.928621:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536244.928629:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536244.928630:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536244.928631:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536244.928632:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584536244.928634:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584536244.928635:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18045 00000100:00080000:4.0:1584536244.928637:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584536244.928643:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536244.928645:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536244.928645:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536244.928646:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536244.928647:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536244.928648:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536244.928649:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536244.928650:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536244.976646:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536244.976651:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536244.976653:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536244.976653:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536244.976656:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536244.976658:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536244.976659:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536244.976661:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536326 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584536244.976670:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078422704 new x1661492078422768 00000800:00020000:4.0:1584536250.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536250.916084:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536250.916088:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536250.916511:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584536250.928598:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536244/real 1584536244] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536331 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584536250.928616:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584536250.928617:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584536250.928618:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584536250.928624:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536250.928625:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536250.928627:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536250.928628:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536250.928629:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536250.928630:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584536250.928631:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584536250.928632:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18050 00000100:00080000:4.0:1584536250.928634:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584536250.928643:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536250.928644:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536250.928645:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536250.928646:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536250.928646:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536250.928647:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536250.928648:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536250.928649:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536250.956125:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536250.956129:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536250.956131:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536250.956131:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536250.956133:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536250.956134:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536250.956136:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536250.956138:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536331 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584536250.956171:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078422768 new x1661492078422800 00000800:00020000:4.0:1584536256.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536256.916081:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536256.916083:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536256.916553:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584536256.928632:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536250/real 1584536250] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536337 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584536256.928641:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584536256.928642:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584536256.928643:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584536256.928649:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536256.928651:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536256.928652:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536256.928653:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536256.928654:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536256.928655:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584536256.928655:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584536256.928657:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18056 00000100:00080000:6.0:1584536256.928659:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584536256.928668:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536256.928670:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536256.928670:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536256.928671:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536256.928673:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536256.928674:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536256.928675:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536256.928676:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536256.947403:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536256.947407:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536256.947409:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536256.947409:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536256.947411:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536256.947413:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536256.947414:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536256.947416:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536337 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584536256.947444:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078422800 new x1661492078422848 00000800:00000100:0.0:1584536263.141995:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536263.142414:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584536263.154461:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536256/real 1584536256] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536343 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584536263.154468:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584536263.154469:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584536263.154471:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584536263.154475:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536263.154477:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536263.154477:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536263.154479:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536263.154479:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536263.154481:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584536263.154481:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584536263.154483:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18062 00000100:00080000:6.0:1584536263.154485:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584536263.154493:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536263.154494:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536263.154495:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536263.154497:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536263.154497:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536263.154498:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536263.154499:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536263.154500:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536263.199517:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536263.199520:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536263.199522:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536263.199523:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536263.199525:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536263.199527:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536263.199528:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536263.199530:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536343 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584536263.199539:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078422848 new x1661492078422880 00000800:00020000:4.0:1584536268.916147:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536268.916150:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536268.916152:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536268.916605:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584536268.928683:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536263/real 1584536263] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536350 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584536268.928691:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584536268.928692:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584536268.928693:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584536268.928709:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536268.928710:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536268.928713:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536268.928714:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536268.928715:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536268.928716:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584536268.928717:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584536268.928718:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18069 00000100:00080000:6.0:1584536268.928720:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584536268.928726:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536268.928728:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536268.928728:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536268.928730:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536268.928730:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536268.928731:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536268.928732:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536268.928733:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536268.973828:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536268.973833:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536268.973834:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536268.973835:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536268.973836:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536268.973838:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536268.973839:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536268.973841:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536350 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584536268.973851:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078422880 new x1661492078422960 00000800:00020000:2.0:1584536274.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536274.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536274.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536274.916491:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584536274.928574:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536268/real 1584536268] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536355 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584536274.928592:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584536274.928593:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584536274.928594:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584536274.928600:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536274.928602:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536274.928602:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536274.928604:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536274.928604:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536274.928605:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584536274.928606:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584536274.928608:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18074 00000100:00080000:6.0:1584536274.928610:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584536274.928619:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536274.928620:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536274.928620:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536274.928621:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536274.928622:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536274.928623:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536274.928624:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536274.928625:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536274.977646:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536274.977650:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536274.977651:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536274.977652:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536274.977653:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536274.977655:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536274.977657:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536274.977658:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536355 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584536274.977668:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078422960 new x1661492078422992 00000800:00020000:4.0:1584536280.916184:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536280.916186:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536280.916188:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536280.916711:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584536280.928787:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536274/real 1584536274] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536361 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584536280.928795:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584536280.928796:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584536280.928797:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584536280.928803:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536280.928805:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536280.928806:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536280.928807:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536280.928807:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536280.928809:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584536280.928810:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584536280.928811:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18080 00000100:00080000:6.0:1584536280.928814:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584536280.928822:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536280.928823:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536280.928824:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536280.928825:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536280.928825:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536280.928827:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536280.928827:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536280.928828:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536280.985589:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536280.985593:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536280.985594:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536280.985595:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536280.985598:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536280.985599:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536280.985601:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536280.985603:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536361 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584536280.985612:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078422992 new x1661492078423024 00000800:00000100:0.0:1584536287.143053:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536287.143482:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584536287.155552:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536280/real 1584536280] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536367 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584536287.155560:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584536287.155561:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584536287.155562:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584536287.155568:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536287.155569:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536287.155570:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536287.155572:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536287.155573:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536287.155574:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584536287.155575:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584536287.155576:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18086 00000100:00080000:6.0:1584536287.155578:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584536287.155587:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536287.155588:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536287.155589:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536287.155590:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536287.155590:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536287.155592:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536287.155592:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584536287.155593:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536287.201953:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536287.201957:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536287.201959:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536287.201959:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536287.201961:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536287.201963:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536287.201964:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536287.201966:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536367 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584536287.201976:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078423024 new x1661492078423072 00000800:00020000:2.0:1584536292.916085:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536292.916088:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536292.916090:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536292.916492:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584536292.928624:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536287/real 1584536287] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536374 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584536292.928634:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584536292.928635:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584536292.928637:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584536292.928657:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536292.928659:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536292.928662:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536292.928663:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536292.928664:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536292.928666:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584536292.928667:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584536292.928669:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18093 00000100:00080000:10.0:1584536292.928671:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584536292.928678:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536292.928679:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536292.928681:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536292.928682:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536292.928683:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536292.928684:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536292.928685:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584536292.928686:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536292.966680:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536292.966686:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536292.966689:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536292.966690:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536292.966692:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536292.966694:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536292.966695:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536292.966697:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536374 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584536292.966727:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078423072 new x1661492078423136 00000800:00020000:2.0:1584536298.916073:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536298.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536298.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536298.916534:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584536298.928623:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536292/real 1584536292] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536379 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584536298.928649:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584536298.928650:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584536298.928652:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584536298.928659:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536298.928661:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536298.928662:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536298.928664:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536298.928664:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536298.928666:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584536298.928667:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584536298.928669:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18098 00000100:00080000:12.0:1584536298.928672:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584536298.928687:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536298.928688:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536298.928689:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536298.928692:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536298.928693:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536298.928705:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536298.928706:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536298.928707:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536298.974366:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536298.974372:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536298.974375:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536298.974376:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536298.974378:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536298.974381:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536298.974382:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536298.974384:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536379 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584536298.974431:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078423136 new x1661492078423184 00000800:00020000:4.0:1584536304.916044:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536304.916047:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536304.916050:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536304.916470:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584536304.928522:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536298/real 1584536298] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536385 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584536304.928533:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584536304.928534:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584536304.928536:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584536304.928545:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536304.928547:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536304.928548:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536304.928549:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536304.928550:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536304.928552:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584536304.928553:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584536304.928555:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18104 00000100:00080000:12.0:1584536304.928558:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584536304.928573:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536304.928575:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536304.928576:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536304.928578:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536304.928578:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536304.928580:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536304.928580:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536304.928582:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536304.962169:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536304.962176:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536304.962178:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536304.962179:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536304.962181:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536304.962184:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536304.962185:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536304.962187:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536385 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584536304.962216:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078423184 new x1661492078423216 00000800:00000100:0.0:1584536311.140672:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536311.141093:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584536311.153163:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536304/real 1584536304] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536391 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584536311.153170:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584536311.153171:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584536311.153173:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584536311.153178:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536311.153180:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536311.153180:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536311.153182:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536311.153182:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536311.153184:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584536311.153184:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584536311.153186:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18110 00000100:00080000:12.0:1584536311.153188:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584536311.153197:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536311.153199:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536311.153200:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536311.153201:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536311.153201:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536311.153203:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536311.153203:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536311.153215:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536311.196002:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536311.196007:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536311.196008:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536311.196009:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536311.196010:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536311.196012:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536311.196014:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536311.196015:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536391 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584536311.196046:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078423216 new x1661492078423248 00000800:00020000:2.0:1584536316.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536316.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536316.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536316.916502:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584536316.928556:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536311/real 1584536311] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536398 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584536316.928565:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584536316.928566:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584536316.928568:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584536316.928585:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536316.928587:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536316.928591:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536316.928592:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536316.928593:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536316.928595:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584536316.928597:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584536316.928598:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18117 00000100:00080000:12.0:1584536316.928601:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584536316.928608:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536316.928609:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536316.928610:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536316.928612:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536316.928613:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536316.928614:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536316.928614:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536316.928615:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536316.972174:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536316.972179:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536316.972181:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536316.972183:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536316.972185:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536316.972187:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536316.972188:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536316.972190:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536398 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584536316.972220:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078423248 new x1661492078423328 00000800:00020000:2.0:1584536322.916090:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536322.916092:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536322.916094:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536322.916555:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584536322.928639:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536316/real 1584536316] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536403 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584536322.928646:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584536322.928648:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584536322.928660:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584536322.928666:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536322.928667:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536322.928668:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536322.928669:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536322.928671:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536322.928672:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584536322.928673:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584536322.928674:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18122 00000100:00080000:12.0:1584536322.928676:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584536322.928685:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536322.928686:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536322.928687:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536322.928688:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536322.928688:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536322.928690:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536322.928690:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536322.928691:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536322.968813:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536322.968821:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536322.968824:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536322.968826:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536322.968829:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536322.968833:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536322.968837:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536322.968841:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536403 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584536322.968891:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078423328 new x1661492078423360 00000800:00020000:4.0:1584536328.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536328.916081:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536328.916083:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536328.916495:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584536328.928547:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536322/real 1584536322] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536409 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584536328.928555:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584536328.928556:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584536328.928557:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584536328.928562:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536328.928564:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536328.928566:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536328.928567:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536328.928568:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536328.928569:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584536328.928570:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584536328.928571:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18128 00000100:00080000:12.0:1584536328.928574:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584536328.928582:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536328.928583:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536328.928584:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536328.928585:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536328.928586:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536328.928587:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536328.928588:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536328.928589:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536328.979959:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536328.979963:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536328.979965:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536328.979966:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536328.979967:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536328.979969:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536328.979971:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536328.979972:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536409 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584536328.980002:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078423360 new x1661492078423408 00000800:00000100:0.0:1584536335.141087:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536335.141600:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584536335.153671:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536328/real 1584536328] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536415 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584536335.153679:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584536335.153680:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584536335.153681:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584536335.153688:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536335.153689:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536335.153690:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536335.153691:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536335.153692:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536335.153693:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584536335.153694:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584536335.153695:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18134 00000100:00080000:12.0:1584536335.153697:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584536335.153705:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536335.153706:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536335.153707:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536335.153708:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536335.153708:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536335.153709:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536335.153710:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536335.153723:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536335.198753:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536335.198757:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536335.198759:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536335.198760:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536335.198762:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536335.198764:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536335.198765:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536335.198767:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536415 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584536335.198796:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078423408 new x1661492078423440 00000800:00020000:2.0:1584536340.916082:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536340.916085:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536340.916087:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536340.916472:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584536340.928524:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536335/real 1584536335] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536422 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584536340.928532:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584536340.928534:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584536340.928535:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584536340.928550:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536340.928551:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536340.928554:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536340.928556:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536340.928556:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536340.928558:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584536340.928558:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584536340.928560:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18141 00000100:00080000:12.0:1584536340.928562:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584536340.928567:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536340.928568:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536340.928569:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536340.928570:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536340.928571:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536340.928572:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536340.928573:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536340.928574:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536340.982428:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536340.982432:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536340.982433:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536340.982434:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536340.982436:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536340.982437:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536340.982439:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536340.982441:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536422 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584536340.982470:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078423440 new x1661492078423504 00000800:00020000:2.0:1584536346.916103:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536346.916109:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536346.916113:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536346.916617:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584536346.928731:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536340/real 1584536340] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536427 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584536346.928741:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584536346.928742:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584536346.928744:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584536346.928751:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536346.928753:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536346.928753:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536346.928754:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536346.928755:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536346.928757:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584536346.928757:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584536346.928759:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18146 00000100:00080000:12.0:1584536346.928761:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584536346.928773:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536346.928774:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536346.928775:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536346.928776:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536346.928777:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536346.928778:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536346.928778:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536346.928780:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536346.979945:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536346.979950:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536346.979952:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536346.979953:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536346.979955:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536346.979957:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536346.979959:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536346.979960:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536427 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584536346.979989:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078423504 new x1661492078423552 00000800:00020000:4.0:1584536352.916109:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536352.916113:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536352.916115:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536352.916540:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584536352.928598:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536346/real 1584536346] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536433 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584536352.928609:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584536352.928610:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584536352.928612:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584536352.928621:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536352.928623:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536352.928624:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536352.928625:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536352.928626:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536352.928628:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584536352.928629:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584536352.928631:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18152 00000100:00080000:12.0:1584536352.928634:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584536352.928649:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536352.928650:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536352.928651:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536352.928652:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536352.928653:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536352.928654:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536352.928656:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536352.928658:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536352.995299:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536352.995305:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536352.995307:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536352.995309:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536352.995311:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536352.995314:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536352.995315:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536352.995317:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536433 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584536352.995347:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078423552 new x1661492078423584 00000800:00000100:0.0:1584536359.141311:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536359.141771:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584536359.153848:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536352/real 1584536352] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536439 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584536359.153857:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584536359.153858:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584536359.153860:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584536359.153866:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536359.153868:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536359.153869:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536359.153870:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536359.153871:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536359.153872:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584536359.153873:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584536359.153874:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18158 00000100:00080000:12.0:1584536359.153877:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584536359.153889:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536359.153890:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536359.153891:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536359.153892:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536359.153894:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536359.153906:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536359.153907:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536359.153908:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536359.199782:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536359.199789:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536359.199791:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536359.199792:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536359.199793:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536359.199796:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536359.199798:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536359.199800:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536439 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584536359.199829:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078423584 new x1661492078423632 00000800:00020000:2.0:1584536364.916048:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536364.916051:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536364.916053:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536364.916470:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584536364.928523:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536359/real 1584536359] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536446 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584536364.928531:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584536364.928532:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584536364.928533:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584536364.928548:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536364.928549:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536364.928553:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536364.928554:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536364.928555:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536364.928556:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584536364.928557:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584536364.928558:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18165 00000100:00080000:12.0:1584536364.928560:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584536364.928566:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536364.928567:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536364.928569:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536364.928570:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536364.928571:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536364.928572:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536364.928573:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536364.928574:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536364.978294:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536364.978298:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536364.978300:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536364.978300:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536364.978302:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536364.978304:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536364.978305:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536364.978307:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536446 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584536364.978336:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078423632 new x1661492078423696 00000800:00020000:4.0:1584536370.916095:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536370.916097:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536370.916099:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536370.916544:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584536370.928597:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536364/real 1584536364] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536451 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584536370.928605:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584536370.928606:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584536370.928607:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584536370.928612:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536370.928614:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536370.928615:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536370.928616:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536370.928616:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536370.928618:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584536370.928618:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584536370.928620:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18170 00000100:00080000:12.0:1584536370.928622:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584536370.928631:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536370.928633:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536370.928633:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536370.928634:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536370.928635:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536370.928636:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536370.928637:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536370.928638:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536370.980852:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536370.980856:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536370.980857:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536370.980858:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536370.980860:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536370.980862:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536370.980863:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536370.980865:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536451 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584536370.980896:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078423696 new x1661492078423728 00000800:00020000:2.0:1584536376.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536376.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536376.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536376.916497:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584536376.928550:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536370/real 1584536370] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536457 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584536376.928558:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584536376.928559:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584536376.928561:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584536376.928567:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536376.928568:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536376.928569:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536376.928570:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536376.928571:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536376.928572:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584536376.928573:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584536376.928576:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18176 00000100:00080000:12.0:1584536376.928578:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584536376.928587:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536376.928588:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536376.928589:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536376.928590:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536376.928590:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536376.928591:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536376.928592:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536376.928593:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536376.986425:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536376.986429:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536376.986431:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536376.986431:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536376.986433:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536376.986435:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536376.986436:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536376.986438:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536457 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584536376.986447:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078423728 new x1661492078423776 00000800:00000100:0.0:1584536383.141476:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536383.141904:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584536383.153953:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536376/real 1584536376] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536463 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584536383.153961:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584536383.153962:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584536383.153963:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584536383.153969:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536383.153970:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536383.153971:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536383.153972:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536383.153973:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536383.153975:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584536383.153976:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584536383.153977:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18182 00000100:00080000:12.0:1584536383.153979:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584536383.153987:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536383.153988:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536383.153989:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536383.153990:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536383.153991:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536383.153992:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536383.153993:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536383.153994:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536383.194340:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536383.194344:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536383.194345:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536383.194346:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536383.194347:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536383.194349:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536383.194351:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536383.194352:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536463 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584536383.194382:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078423776 new x1661492078423808 00000800:00020000:2.0:1584536388.916064:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536388.916066:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536388.916068:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536388.916462:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584536388.928515:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536383/real 1584536383] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536470 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584536388.928523:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584536388.928524:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584536388.928525:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584536388.928541:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536388.928542:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536388.928545:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536388.928548:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536388.928548:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536388.928550:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584536388.928550:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584536388.928552:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18189 00000100:00080000:12.0:1584536388.928554:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584536388.928559:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536388.928561:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536388.928561:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536388.928563:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536388.928563:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536388.928564:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536388.928565:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536388.928566:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536388.988359:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536388.988363:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536388.988364:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536388.988365:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536388.988367:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536388.988369:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536388.988371:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536388.988373:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536470 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584536388.988381:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078423808 new x1661492078423888 00000800:00020000:2.0:1584536394.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536394.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536394.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536394.916517:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584536394.928570:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536388/real 1584536388] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536475 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584536394.928578:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584536394.928579:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584536394.928580:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584536394.928586:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536394.928588:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536394.928589:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536394.928590:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536394.928591:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536394.928592:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584536394.928593:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584536394.928594:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18194 00000100:00080000:12.0:1584536394.928596:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584536394.928605:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536394.928606:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536394.928607:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536394.928608:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536394.928608:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536394.928610:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536394.928610:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536394.928611:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536394.982563:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536394.982567:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536394.982569:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536394.982569:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536394.982571:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536394.982572:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536394.982574:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536394.982576:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536475 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584536394.982584:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078423888 new x1661492078423920 00000800:00020000:2.0:1584536400.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536400.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536400.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536400.916583:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584536400.928638:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536394/real 1584536394] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536481 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584536400.928646:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584536400.928647:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584536400.928649:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584536400.928655:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536400.928657:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536400.928657:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536400.928659:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536400.928659:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536400.928661:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584536400.928661:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584536400.928663:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18200 00000100:00080000:12.0:1584536400.928665:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584536400.928673:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536400.928674:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536400.928675:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536400.928676:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536400.928677:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536400.928678:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536400.928678:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536400.928680:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536400.974114:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536400.974118:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536400.974120:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536400.974120:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536400.974122:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536400.974123:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536400.974125:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536400.974127:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536481 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584536400.974135:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078423920 new x1661492078423952 00000800:00000100:0.0:1584536407.141983:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536407.142425:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584536407.154487:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536400/real 1584536400] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536487 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584536407.154502:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584536407.154503:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584536407.154505:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584536407.154512:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536407.154514:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536407.154515:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536407.154516:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536407.154517:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536407.154519:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584536407.154520:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584536407.154522:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18206 00000100:00080000:12.0:1584536407.154524:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584536407.154538:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536407.154539:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536407.154540:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536407.154541:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536407.154542:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536407.154543:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536407.154544:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536407.154545:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536407.206675:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536407.206681:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536407.206683:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536407.206684:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536407.206687:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536407.206689:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536407.206690:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536407.206692:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536487 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584536407.206723:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078423952 new x1661492078424000 00000800:00020000:6.0:1584536412.916055:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584536412.916058:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584536412.916061:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536412.916490:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584536412.928546:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536407/real 1584536407] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536494 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584536412.928555:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584536412.928557:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584536412.928558:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584536412.928579:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536412.928581:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536412.928585:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536412.928586:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536412.928587:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536412.928588:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584536412.928589:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584536412.928591:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18213 00000100:00080000:12.0:1584536412.928594:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584536412.928601:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536412.928602:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536412.928603:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536412.928604:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536412.928605:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536412.928606:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536412.928608:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536412.928609:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536412.969531:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536412.969538:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536412.969540:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536412.969541:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536412.969542:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536412.969545:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536412.969546:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536412.969549:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536494 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584536412.969557:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078424000 new x1661492078424064 00000002:00080000:2.0:1584536416.433681:0:163139:0:(mdc_request.c:903:mdc_close()) @@@ matched open req@ffff8d3396acec00 x1661492078424080/t87758643298(87758643298) o101->public1-MDT0000-mdc-ffff8d3ae8ab1800@10.10.2.11@o2ib:12/10 lens 576/616 e 0 to 0 dl 1584536423 ref 1 fl Complete:RP/4/ffffffff rc 0/-1 00000002:00080000:12.0:1584536416.757248:0:163141:0:(mdc_request.c:903:mdc_close()) @@@ matched open req@ffff8d3396acf500 x1661492078424112/t87758643299(87758643299) o101->public1-MDT0000-mdc-ffff8d3ae8ab1800@10.10.2.11@o2ib:12/10 lens 576/616 e 0 to 0 dl 1584536423 ref 1 fl Complete:RP/4/ffffffff rc 0/-1 00000800:00020000:0.0:1584536418.916034:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584536418.916037:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584536418.916039:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536418.916690:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584536418.928749:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536412/real 1584536412] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536499 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584536418.928758:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584536418.928760:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584536418.928762:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584536418.928769:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536418.928771:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536418.928772:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536418.928774:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536418.928774:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536418.928776:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584536418.928777:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584536418.928779:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18218 00000100:00080000:12.0:1584536418.928782:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584536418.928792:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536418.928794:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536418.928795:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536418.928797:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536418.928797:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536418.928799:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536418.928799:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536418.928801:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536418.986501:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536418.986507:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536418.986509:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536418.986510:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536418.986512:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536418.986515:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536418.986516:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536418.986518:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536499 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584536418.986548:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078424064 new x1661492078424208 00000800:00020000:2.0:1584536424.916041:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536424.916044:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536424.916046:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536424.916476:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584536424.928529:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536418/real 1584536418] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536505 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584536424.928539:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584536424.928541:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584536424.928543:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584536424.928550:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536424.928552:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536424.928553:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536424.928554:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536424.928555:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536424.928557:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584536424.928558:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584536424.928560:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18224 00000100:00080000:12.0:1584536424.928563:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584536424.928577:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536424.928579:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536424.928580:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536424.928581:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536424.928582:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536424.928583:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536424.928584:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536424.928585:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536424.981672:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536424.981679:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536424.981681:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536424.981682:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536424.981685:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536424.981688:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536424.981689:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536424.981691:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536505 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584536424.981720:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078424208 new x1661492078424240 00000800:00000100:0.0:1584536431.139042:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536431.139498:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584536431.151550:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536424/real 1584536424] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536511 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584536431.151558:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584536431.151560:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584536431.151561:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584536431.151567:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536431.151569:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536431.151570:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536431.151571:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536431.151572:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536431.151573:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584536431.151574:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584536431.151576:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18230 00000100:00080000:12.0:1584536431.151579:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584536431.151590:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536431.151591:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536431.151592:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536431.151593:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536431.151594:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536431.151596:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536431.151596:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536431.151598:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536431.204177:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584536431.204182:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584536431.204184:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584536431.204185:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584536431.204187:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584536431.204189:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584536431.204191:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584536431.204192:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536511 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584536431.204221:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078424240 new x1661492078424272 00000800:00020000:2.0:1584536436.916073:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536436.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536436.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536436.916485:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584536436.928544:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536431/real 1584536431] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536518 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584536436.928552:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584536436.928553:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584536436.928554:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584536436.928569:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536436.928571:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536436.928576:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536436.928578:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536436.928578:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536436.928580:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584536436.928581:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584536436.928583:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18237 00000100:00080000:12.0:1584536436.928585:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584536436.928591:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536436.928592:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536436.928593:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536436.928594:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536436.928595:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536436.928596:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536436.928597:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584536436.928598:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536436.943269:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584536436.943274:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584536436.943275:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584536436.943276:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584536436.943278:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584536436.943281:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584536436.943282:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584536436.943284:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536518 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536436.943293:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078424272 new x1661492078424352 00000800:00020000:2.0:1584536442.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536442.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536442.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536442.916525:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536442.928578:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536436/real 1584536436] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536523 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536442.928586:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536442.928587:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536442.928588:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536442.928594:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536442.928596:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536442.928597:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536442.928598:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536442.928599:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536442.928600:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536442.928601:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536442.928602:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18242 00000100:00080000:16.0:1584536442.928604:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536442.928614:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536442.928616:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536442.928616:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536442.928617:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536442.928618:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536442.928619:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536442.928621:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536442.928622:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536442.998020:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584536442.998024:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584536442.998025:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584536442.998044:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584536442.998045:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584536442.998047:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584536442.998049:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584536442.998051:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536523 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536442.998062:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078424352 new x1661492078424384 00000800:00020000:2.0:1584536448.916062:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536448.916064:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536448.916066:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536448.916527:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536448.928582:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536442/real 1584536442] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536529 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536448.928590:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536448.928591:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536448.928592:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536448.928598:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536448.928600:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536448.928601:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536448.928602:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536448.928603:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536448.928604:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536448.928605:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536448.928607:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18248 00000100:00080000:16.0:1584536448.928609:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536448.928618:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536448.928620:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536448.928620:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536448.928621:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536448.928623:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536448.928624:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536448.928625:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536448.928626:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536448.971394:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584536448.971398:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584536448.971400:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584536448.971401:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584536448.971402:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584536448.971404:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584536448.971405:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584536448.971407:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536529 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536448.971416:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078424384 new x1661492078424432 00000800:00000100:0.0:1584536455.139446:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536455.139872:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536455.151930:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536448/real 1584536448] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536535 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536455.151940:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536455.151942:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536455.151944:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536455.151952:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536455.151954:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536455.151955:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536455.151956:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536455.151957:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536455.151959:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536455.151960:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536455.151962:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18254 00000100:00080000:16.0:1584536455.151965:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536455.151979:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536455.151981:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536455.151983:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536455.151985:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536455.151985:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536455.151987:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536455.151987:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536455.151989:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536455.200009:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584536455.200016:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584536455.200019:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584536455.200020:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584536455.200023:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584536455.200025:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584536455.200035:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584536455.200037:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536535 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536455.200046:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078424432 new x1661492078424464 00000800:00020000:2.0:1584536460.916099:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536460.916102:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536460.916104:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536460.916521:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536460.928597:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536455/real 1584536455] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536542 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536460.928605:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536460.928606:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536460.928607:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536460.928622:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536460.928624:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536460.928629:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536460.928631:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536460.928631:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536460.928633:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536460.928633:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536460.928635:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18261 00000100:00080000:16.0:1584536460.928637:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536460.928644:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536460.928645:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536460.928646:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536460.928647:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536460.928648:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536460.928649:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536460.928650:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536460.928651:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536460.968227:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584536460.968231:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584536460.968233:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584536460.968234:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584536460.968235:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584536460.968237:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584536460.968239:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584536460.968241:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536542 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536460.968270:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078424464 new x1661492078424528 00000800:00020000:2.0:1584536466.916056:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536466.916059:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536466.916062:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536466.916486:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536466.928544:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536460/real 1584536460] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536547 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536466.928555:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536466.928557:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536466.928559:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536466.928568:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536466.928570:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536466.928571:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536466.928572:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536466.928573:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536466.928575:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536466.928576:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536466.928579:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18266 00000100:00080000:16.0:1584536466.928582:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536466.928597:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536466.928599:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536466.928599:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536466.928601:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536466.928601:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536466.928603:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536466.928603:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536466.928605:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536466.971427:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584536466.971435:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584536466.971438:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584536466.971439:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584536466.971441:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584536466.971443:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584536466.971445:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584536466.971447:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536547 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536466.971456:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078424528 new x1661492078424576 00000800:00020000:4.0:1584536472.916081:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536472.916085:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536472.916087:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536472.916481:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536472.928563:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536466/real 1584536466] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536553 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536472.928573:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536472.928575:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536472.928577:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536472.928585:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536472.928587:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536472.928588:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536472.928589:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536472.928590:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536472.928593:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536472.928594:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536472.928596:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18272 00000100:00080000:16.0:1584536472.928599:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536472.928615:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536472.928616:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536472.928617:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536472.928618:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536472.928619:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536472.928620:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536472.928621:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536472.928622:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536472.936672:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584536472.936679:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584536472.936681:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584536472.936682:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584536472.936684:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584536472.936687:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584536472.936689:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584536472.936691:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536553 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536472.936722:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078424576 new x1661492078424608 00000800:00000100:0.0:1584536479.142000:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536479.142513:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536479.154568:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536472/real 1584536472] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536559 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536479.154577:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536479.154579:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536479.154581:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536479.154588:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536479.154590:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536479.154591:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536479.154594:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536479.154594:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536479.154596:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536479.154597:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536479.154599:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18278 00000100:00080000:16.0:1584536479.154602:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536479.154616:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536479.154617:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536479.154618:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536479.154619:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536479.154619:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536479.154621:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536479.154622:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536479.154623:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584536479.162718:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584536479.162726:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584536479.162728:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584536479.162730:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584536479.162732:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584536479.162734:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584536479.162736:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584536479.162738:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536559 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536479.162747:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078424608 new x1661492078424656 00000800:00020000:4.0:1584536484.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536484.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536484.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536484.916557:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536484.928634:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536479/real 1584536479] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536566 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536484.928642:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536484.928643:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536484.928644:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536484.928659:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536484.928662:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536484.928667:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536484.928669:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536484.928669:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536484.928671:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536484.928671:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536484.928673:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18285 00000100:00080000:16.0:1584536484.928675:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536484.928680:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536484.928681:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536484.928682:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536484.928683:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536484.928684:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536484.928685:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536484.928685:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536484.928687:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584536484.936423:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584536484.936427:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584536484.936428:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584536484.936429:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584536484.936431:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584536484.936433:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584536484.936434:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584536484.936436:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536566 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536484.936465:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078424656 new x1661492078424720 00000800:00020000:4.0:1584536490.916092:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536490.916094:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536490.916096:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536490.916616:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536490.928699:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536484/real 1584536484] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536571 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536490.928707:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536490.928708:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536490.928721:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536490.928727:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536490.928728:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536490.928729:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536490.928730:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536490.928731:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536490.928732:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536490.928733:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536490.928734:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18290 00000100:00080000:16.0:1584536490.928737:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536490.928745:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536490.928746:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536490.928747:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536490.928748:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536490.928748:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536490.928749:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536490.928750:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536490.928751:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584536490.936659:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584536490.936667:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584536490.936670:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584536490.936672:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584536490.936675:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584536490.936679:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584536490.936683:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584536490.936687:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536571 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536490.936734:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078424720 new x1661492078424752 00000800:00020000:2.0:1584536496.916073:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536496.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536496.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536496.916562:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536496.928639:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536490/real 1584536490] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536577 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536496.928652:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536496.928653:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536496.928654:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536496.928660:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536496.928662:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536496.928663:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536496.928664:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536496.928664:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536496.928666:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536496.928666:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536496.928668:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18296 00000100:00080000:16.0:1584536496.928670:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536496.928679:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536496.928680:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536496.928680:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536496.928682:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536496.928682:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536496.928683:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536496.928684:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536496.928685:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584536496.936725:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584536496.936729:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584536496.936731:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584536496.936732:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584536496.936733:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584536496.936735:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584536496.936736:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584536496.936738:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536577 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536496.936767:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078424752 new x1661492078424800 00000800:00000100:0.0:1584536503.142149:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536503.142555:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536503.154605:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536496/real 1584536496] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536583 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536503.154613:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536503.154614:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536503.154615:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536503.154621:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536503.154622:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536503.154623:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536503.154624:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536503.154625:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536503.154626:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536503.154627:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536503.154628:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18302 00000100:00080000:16.0:1584536503.154630:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536503.154639:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536503.154640:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536503.154641:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536503.154642:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536503.154642:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536503.154644:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536503.154645:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536503.154647:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584536503.162464:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584536503.162468:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584536503.162469:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584536503.162470:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584536503.162471:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584536503.162473:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584536503.162475:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584536503.162476:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536583 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536503.162485:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078424800 new x1661492078424832 00000800:00020000:2.0:1584536508.916070:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536508.916073:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536508.916074:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536508.916485:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536508.928560:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536503/real 1584536503] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536590 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536508.928568:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536508.928569:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536508.928571:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536508.928585:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536508.928587:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536508.928592:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536508.928593:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536508.928594:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536508.928595:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536508.928596:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536508.928597:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18309 00000100:00080000:16.0:1584536508.928599:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536508.928605:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536508.928606:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536508.928607:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536508.928608:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536508.928609:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536508.928610:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536508.928611:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536508.928612:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536508.936644:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584536508.936649:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584536508.936650:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584536508.936651:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584536508.936652:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584536508.936655:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584536508.936657:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584536508.936658:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536590 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536508.936687:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078424832 new x1661492078424912 00000800:00020000:4.0:1584536514.916070:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536514.916073:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536514.916074:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536514.916610:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536514.928692:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536508/real 1584536508] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536595 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536514.928710:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536514.928711:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536514.928713:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536514.928719:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536514.928720:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536514.928721:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536514.928722:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536514.928723:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536514.928724:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536514.928725:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536514.928727:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18314 00000100:00080000:16.0:1584536514.928729:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536514.928737:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536514.928738:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536514.928740:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536514.928741:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536514.928742:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536514.928743:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536514.928744:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536514.928745:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584536514.936779:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584536514.936783:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584536514.936784:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584536514.936785:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584536514.936786:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584536514.936789:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584536514.936790:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584536514.936792:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536595 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536514.936826:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078424912 new x1661492078424944 00000800:00020000:2.0:1584536520.916066:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536520.916069:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536520.916071:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536520.916479:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536520.928557:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536514/real 1584536514] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536601 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536520.928565:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536520.928566:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536520.928567:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536520.928573:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536520.928575:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536520.928576:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536520.928577:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536520.928578:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536520.928579:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536520.928579:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536520.928581:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18320 00000100:00080000:16.0:1584536520.928583:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536520.928593:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536520.928594:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536520.928595:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536520.928596:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536520.928597:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536520.928598:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536520.928599:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536520.928600:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536520.936473:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584536520.936477:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584536520.936479:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584536520.936480:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584536520.936481:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584536520.936483:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584536520.936484:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584536520.936486:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536601 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536520.936516:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078424944 new x1661492078424976 00000800:00000100:0.0:1584536527.142043:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536527.142542:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536527.154662:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536520/real 1584536520] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536607 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536527.154672:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536527.154673:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536527.154676:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536527.154683:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536527.154685:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536527.154686:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536527.154687:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536527.154688:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536527.154689:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536527.154690:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536527.154693:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18326 00000100:00080000:16.0:1584536527.154696:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536527.154710:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536527.154723:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536527.154724:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536527.154725:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536527.154726:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536527.154727:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536527.154728:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536527.154729:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536527.162636:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584536527.162642:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584536527.162644:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584536527.162645:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584536527.162647:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584536527.162650:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584536527.162651:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584536527.162653:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536607 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536527.162688:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078424976 new x1661492078425024 00000800:00020000:2.0:1584536532.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536532.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536532.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536532.916504:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536532.928585:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536527/real 1584536527] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536614 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536532.928595:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536532.928597:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536532.928599:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536532.928621:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536532.928623:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536532.928628:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536532.928629:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536532.928630:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536532.928633:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536532.928634:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536532.928637:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18333 00000100:00080000:16.0:1584536532.928640:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536532.928648:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536532.928649:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536532.928650:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536532.928651:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536532.928652:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536532.928653:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536532.928654:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536532.928655:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536532.936796:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584536532.936803:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584536532.936805:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584536532.936807:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584536532.936808:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584536532.936811:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584536532.936813:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584536532.936815:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536614 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536532.936845:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078425024 new x1661492078425088 00000800:00020000:4.0:1584536538.916098:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536538.916104:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536538.916108:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536538.916641:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536538.928727:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536532/real 1584536532] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536619 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536538.928746:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536538.928747:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536538.928748:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536538.928754:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536538.928756:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536538.928757:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536538.928759:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536538.928759:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536538.928761:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536538.928761:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536538.928763:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18338 00000100:00080000:16.0:1584536538.928765:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536538.928776:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536538.928778:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536538.928778:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536538.928780:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536538.928780:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536538.928781:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536538.928782:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536538.928783:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536538.936940:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584536538.936945:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584536538.936947:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584536538.936948:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584536538.936950:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584536538.936952:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584536538.936954:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584536538.936956:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536619 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536538.936984:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078425088 new x1661492078425136 00000800:00020000:0.0:1584536544.916102:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584536544.916105:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584536544.916107:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536544.916608:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536544.928682:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536538/real 1584536538] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536625 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536544.928690:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536544.928691:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536544.928693:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536544.928699:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536544.928701:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536544.928702:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536544.928703:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536544.928704:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536544.928705:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536544.928706:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536544.928707:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18344 00000100:00080000:16.0:1584536544.928710:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536544.928718:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536544.928719:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536544.928720:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536544.928721:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536544.928722:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536544.928723:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536544.928724:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536544.928725:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536544.936715:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584536544.936719:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584536544.936720:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584536544.936721:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584536544.936722:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584536544.936724:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584536544.936725:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584536544.936727:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536625 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536544.936757:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078425136 new x1661492078425168 00000800:00000100:0.0:1584536551.140867:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536551.141325:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536551.153398:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536544/real 1584536544] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536631 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536551.197208:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536551.197209:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536551.197211:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536551.197215:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536551.197217:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536551.197217:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536551.197219:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536551.197219:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536551.197220:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536551.197221:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536551.197222:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18350 00000100:00080000:16.0:1584536551.197224:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536551.197232:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536551.197233:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536551.197234:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536551.197235:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536551.197235:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536551.197237:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536551.197237:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536551.197238:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536551.206717:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584536551.206721:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584536551.206722:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584536551.206723:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584536551.206724:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584536551.206726:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584536551.206728:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584536551.206729:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536631 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536551.206759:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078425168 new x1661492078425200 00000800:00020000:0.0:1584536556.916097:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584536556.916100:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584536556.916102:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536556.916513:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536556.928592:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536551/real 1584536551] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536638 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536556.928604:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536556.928605:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536556.928607:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536556.928621:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536556.928623:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536556.928628:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536556.928629:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536556.928630:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536556.928631:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536556.928632:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536556.928633:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18357 00000100:00080000:16.0:1584536556.928635:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536556.928641:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536556.928642:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536556.928643:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536556.928644:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536556.928645:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536556.928646:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536556.928646:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536556.928647:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536556.990492:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584536556.990497:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584536556.990498:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584536556.990499:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584536556.990500:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584536556.990502:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584536556.990504:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584536556.990505:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536638 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536556.990534:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078425200 new x1661492078425280 00000800:00020000:2.0:1584536562.916099:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536562.916105:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536562.916109:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536562.916543:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536562.928632:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536556/real 1584536556] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536643 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536562.928651:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536562.928652:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536562.928653:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536562.928658:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536562.928659:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536562.928660:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536562.928661:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536562.928662:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536562.928663:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536562.928664:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536562.928665:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18362 00000100:00080000:16.0:1584536562.928667:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536562.928675:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536562.928677:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536562.928677:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536562.928678:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536562.928679:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536562.928680:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536562.928681:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536562.928683:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536562.954553:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584536562.954557:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584536562.954559:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584536562.954560:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584536562.954561:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584536562.954563:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584536562.954564:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584536562.954566:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536643 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536562.954597:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078425280 new x1661492078425328 00000800:00020000:4.0:1584536568.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536568.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536568.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536568.916498:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536568.928574:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536562/real 1584536562] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536649 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536568.928582:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536568.928583:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536568.928584:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536568.928590:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536568.928592:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536568.928593:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536568.928594:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536568.928595:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536568.928596:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536568.928596:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536568.928598:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18368 00000100:00080000:16.0:1584536568.928600:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536568.928609:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536568.928610:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536568.928611:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536568.928612:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536568.928614:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536568.928615:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536568.928616:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536568.928617:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536568.956300:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584536568.956304:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584536568.956305:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584536568.956306:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584536568.956308:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584536568.956310:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584536568.956311:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584536568.956313:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536649 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536568.956322:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078425328 new x1661492078425376 00000800:00020000:6.0:1584536574.916108:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584536574.916114:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584536574.916118:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536574.916542:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536574.928633:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536568/real 1584536568] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536655 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536574.928652:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536574.928653:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536574.928654:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536574.928660:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536574.928661:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536574.928662:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536574.928663:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536574.928664:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536574.928665:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536574.928666:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536574.928667:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18374 00000100:00080000:16.0:1584536574.928669:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536574.928677:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536574.928678:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536574.928680:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536574.928681:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536574.928682:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536574.928683:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536574.928683:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536574.928696:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536574.973124:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584536574.973128:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584536574.973130:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584536574.973130:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584536574.973132:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584536574.973133:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584536574.973135:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584536574.973136:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536655 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536574.973170:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078425376 new x1661492078425408 00000800:00020000:4.0:1584536580.916055:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536580.916057:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536580.916059:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536580.916662:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536580.928742:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536574/real 1584536574] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536661 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536580.928750:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536580.928751:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536580.928752:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536580.928767:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536580.928769:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536580.928772:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536580.928774:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536580.928774:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536580.928776:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536580.928776:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536580.928778:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18380 00000100:00080000:16.0:1584536580.928780:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536580.928786:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536580.928788:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536580.928789:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536580.928790:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536580.928790:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536580.928792:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536580.928792:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536580.928793:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536580.982589:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584536580.982593:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584536580.982594:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584536580.982595:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584536580.982596:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584536580.982598:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584536580.982599:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584536580.982601:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536661 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536580.982632:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078425408 new x1661492078425472 00000800:00020000:2.0:1584536586.916093:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536586.916096:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536586.916097:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536586.916580:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536586.928665:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536580/real 1584536580] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536667 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536586.928684:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536586.928685:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536586.928687:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536586.928692:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536586.928694:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536586.928695:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536586.928696:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536586.928697:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536586.928698:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536586.928698:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536586.928702:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18386 00000100:00080000:16.0:1584536586.928704:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536586.928713:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536586.928714:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536586.928715:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536586.928716:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536586.928717:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536586.928718:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536586.928718:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536586.928720:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536586.985085:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584536586.985089:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584536586.985090:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584536586.985091:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584536586.985092:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584536586.985094:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584536586.985096:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584536586.985097:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536667 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536586.985132:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078425472 new x1661492078425520 00000800:00020000:0.0:1584536592.916037:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584536592.916040:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584536592.916043:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536592.916434:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536592.928514:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536586/real 1584536586] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536673 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536592.928523:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536592.928524:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536592.928526:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536592.928534:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536592.928536:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536592.928537:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536592.928539:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536592.928539:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536592.928542:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536592.928543:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536592.928545:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18392 00000100:00080000:16.0:1584536592.928547:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536592.928559:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536592.928561:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536592.928562:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536592.928563:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536592.928564:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536592.928565:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536592.928566:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536592.928567:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536592.959516:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584536592.959521:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584536592.959523:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584536592.959524:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584536592.959526:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584536592.959528:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584536592.959529:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584536592.959531:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536673 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536592.959561:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078425520 new x1661492078425552 00000800:00020000:2.0:1584536598.916042:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536598.916045:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536598.916047:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536598.916523:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536598.928604:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536592/real 1584536592] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536679 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536598.928624:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536598.928626:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536598.928628:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536598.928636:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536598.928638:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536598.928639:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536598.928641:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536598.928642:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536598.928643:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536598.928644:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536598.928646:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18398 00000100:00080000:16.0:1584536598.928648:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536598.928661:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536598.928662:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536598.928663:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536598.928664:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536598.928665:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536598.928666:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536598.928667:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536598.928668:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536598.975497:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584536598.975503:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584536598.975505:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584536598.975506:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584536598.975508:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584536598.975510:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584536598.975512:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584536598.975514:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536679 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536598.975548:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078425552 new x1661492078425600 00000800:00020000:2.0:1584536604.916070:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536604.916073:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536604.916074:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536604.916459:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536604.928539:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536598/real 1584536598] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536685 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536604.928548:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536604.928549:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536604.928551:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536604.928567:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536604.928570:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536604.928574:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536604.928575:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536604.928576:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536604.928577:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536604.928578:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536604.928580:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18404 00000100:00080000:16.0:1584536604.928582:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536604.928588:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536604.928589:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536604.928590:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536604.928591:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536604.928592:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536604.928593:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536604.928594:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536604.928595:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536605.404123:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584536605.404128:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584536605.404129:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584536605.404130:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584536605.404131:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584536605.404134:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584536605.404136:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584536605.404138:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536685 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536605.404167:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078425600 new x1661492078425664 00000800:00020000:2.0:1584536610.916100:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536610.916103:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536610.916105:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536610.916774:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536610.928851:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536605/real 1584536605] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536692 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536610.928858:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536610.928859:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536610.928862:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536610.928868:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536610.928869:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536610.928870:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536610.928871:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536610.928872:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536610.928873:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536610.928874:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536610.928875:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18410 00000100:00080000:16.0:1584536610.928877:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536610.928887:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536610.928888:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536610.928889:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536610.928890:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536610.928890:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536610.928892:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536610.928892:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536610.928893:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536610.996196:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584536610.996200:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584536610.996202:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584536610.996203:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584536610.996204:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584536610.996206:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584536610.996207:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584536610.996209:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536692 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536610.996238:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078425664 new x1661492078425696 00000800:00020000:2.0:1584536616.916052:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536616.916054:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536616.916056:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536616.916472:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536616.928543:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536610/real 1584536610] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536697 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536616.928556:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536616.928557:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536616.928558:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536616.928564:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536616.928566:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536616.928567:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536616.928568:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536616.928569:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536616.928570:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536616.928571:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536616.928572:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18416 00000100:00080000:16.0:1584536616.928574:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536616.928583:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536616.928585:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536616.928585:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536616.928586:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536616.928587:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536616.928588:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536616.928589:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536616.928590:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536617.004730:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584536617.004734:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584536617.004736:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584536617.004737:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584536617.004738:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584536617.004740:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584536617.004741:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584536617.004743:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536697 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584536617.004774:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078425696 new x1661492078425744 00000800:00020000:2.0:1584536622.916070:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536622.916073:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536622.916075:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536622.916459:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584536622.928510:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536617/real 1584536617] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536704 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584536622.928518:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584536622.953900:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584536622.953901:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584536622.953906:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536622.953907:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536622.953908:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536622.953909:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536622.953910:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536622.953911:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584536622.953912:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584536622.953913:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18422 00000100:00080000:16.0:1584536622.953915:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584536622.953924:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536622.953925:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536622.953925:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536622.953927:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536622.953927:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536622.953928:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536622.953930:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584536622.953931:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536623.010290:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584536623.010294:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584536623.010296:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584536623.010296:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584536623.010298:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584536623.010300:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584536623.010301:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584536623.028096:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536704 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584536623.028106:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078425744 new x1661492078425776 00000800:00020000:6.0:1584536628.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584536628.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584536628.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536628.916525:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584536628.928578:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536623/real 1584536623] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536710 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584536628.928586:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584536628.928587:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584536628.928588:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584536628.928603:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536628.928605:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536628.928614:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536628.928616:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536628.928617:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536628.928618:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584536628.928619:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584536628.928621:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18428 00000100:00080000:18.0:1584536628.928623:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584536628.928628:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536628.928629:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536628.928630:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536628.928631:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536628.928631:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536628.928633:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536628.928633:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536628.928634:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536628.975865:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584536628.975869:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584536628.975871:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584536628.975872:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584536628.975873:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584536628.975875:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584536628.975877:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584536628.975878:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536710 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584536628.975887:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078425776 new x1661492078425856 00000800:00020000:2.0:1584536634.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536634.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536634.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536634.916495:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584536634.928546:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536628/real 1584536628] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536715 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584536634.928553:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584536634.928554:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584536634.928556:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584536634.928560:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536634.928562:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536634.928564:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536634.928565:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536634.928565:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536634.928567:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584536634.928567:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584536634.928569:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18434 00000100:00080000:18.0:1584536634.928571:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584536634.928579:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536634.928580:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536634.928581:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536634.928582:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536634.928583:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536634.928584:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536634.928584:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536634.928586:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536634.989670:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584536634.989674:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584536634.989676:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584536634.989677:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584536634.989678:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584536634.989680:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584536634.989681:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584536634.989683:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536715 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584536634.989692:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078425856 new x1661492078425888 00000800:00020000:2.0:1584536640.916055:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536640.916058:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536640.916060:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536640.916509:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584536640.928562:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536634/real 1584536634] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536721 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584536640.928570:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584536640.928571:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584536640.928572:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584536640.928578:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536640.928580:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536640.928581:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536640.928582:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536640.928582:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536640.928584:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584536640.928584:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584536640.928586:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18440 00000100:00080000:18.0:1584536640.928588:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584536640.928596:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536640.928597:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536640.928598:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536640.928599:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536640.928600:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536640.928601:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536640.928601:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536640.928603:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584536640.972444:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584536640.972448:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584536640.972449:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584536640.972450:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584536640.972451:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584536640.972453:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584536640.972455:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584536640.972457:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536721 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584536640.972465:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078425888 new x1661492078425920 00000800:00020000:4.0:1584536646.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536646.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536646.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536646.916491:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584536646.928560:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536640/real 1584536640] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536727 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584536646.928568:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584536646.928570:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584536646.928571:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584536646.928576:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536646.928578:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536646.928578:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536646.928580:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536646.928580:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536646.928582:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584536646.928582:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584536646.928584:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18446 00000100:00080000:18.0:1584536646.928586:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584536646.928596:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536646.928597:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536646.928598:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536646.928599:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536646.928600:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536646.928601:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536646.928601:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536646.928603:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536646.974704:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584536646.974709:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584536646.974711:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584536646.974712:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584536646.974714:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584536646.974716:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584536646.974717:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584536646.974719:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536727 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584536646.974728:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078425920 new x1661492078425968 00000800:00020000:4.0:1584536652.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536652.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536652.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536652.916577:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584536652.928632:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536646/real 1584536646] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536733 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584536652.928642:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584536652.928643:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584536652.928645:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584536652.928665:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536652.928667:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536652.928671:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536652.928672:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536652.928673:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536652.928675:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584536652.928676:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584536652.928678:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18452 00000100:00080000:18.0:1584536652.928680:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584536652.928687:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536652.928688:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536652.928689:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536652.928690:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536652.928691:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536652.928692:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536652.928693:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536652.928694:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536652.982175:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584536652.982180:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584536652.982183:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584536652.982184:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584536652.982185:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584536652.982187:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584536652.982189:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584536652.982191:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536733 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584536652.982204:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078425968 new x1661492078426032 00000800:00020000:4.0:1584536658.916050:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536658.916053:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536658.916055:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536658.916500:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584536658.928555:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536652/real 1584536652] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536739 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584536658.928564:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584536658.928566:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584536658.928567:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584536658.928575:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536658.928577:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536658.928577:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536658.928579:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536658.928579:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536658.928581:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584536658.928582:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584536658.928584:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18458 00000100:00080000:18.0:1584536658.928587:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584536658.928599:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536658.928600:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536658.928601:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536658.928602:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536658.928602:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536658.928604:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536658.928605:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536658.928607:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536658.996395:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584536658.996401:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584536658.996403:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584536658.996404:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584536658.996406:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584536658.996409:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584536658.996410:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584536658.996412:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536739 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584536658.996421:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078426032 new x1661492078426080 00000800:00020000:2.0:1584536664.916081:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536664.916084:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536664.916086:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536664.916593:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584536664.928653:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536658/real 1584536658] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536745 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584536664.928661:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584536664.928662:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584536664.928663:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584536664.928669:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536664.928671:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536664.928672:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536664.928673:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536664.928673:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536664.928675:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584536664.928675:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584536664.928677:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18464 00000100:00080000:18.0:1584536664.928679:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584536664.928688:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536664.928689:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536664.928690:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536664.928691:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536664.928693:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536664.928694:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536664.928694:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536664.928695:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536664.986670:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584536664.986673:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584536664.986675:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584536664.986676:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584536664.986677:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584536664.986679:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584536664.986680:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584536664.986682:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536745 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584536664.986694:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078426080 new x1661492078426112 00000800:00020000:0.0:1584536670.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584536670.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584536670.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536670.916490:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584536670.928540:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536664/real 1584536664] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536751 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584536670.928548:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584536670.928549:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584536670.928551:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584536670.928556:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536670.928558:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536670.928559:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536670.928560:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536670.928561:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536670.928562:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584536670.928563:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584536670.928564:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18470 00000100:00080000:18.0:1584536670.928566:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584536670.928575:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536670.928576:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536670.928577:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536670.928579:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536670.928579:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536670.928580:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536670.928581:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536670.928582:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536670.971607:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584536670.971611:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584536670.971612:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584536670.971613:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584536670.971615:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584536670.971616:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584536670.971618:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584536670.971620:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536751 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584536670.971650:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078426112 new x1661492078426144 00000800:00000100:0.0:1584536677.141813:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536677.142220:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584536677.154270:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536670/real 1584536670] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536757 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584536677.154278:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584536677.154279:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584536677.154280:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584536677.154295:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536677.154297:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536677.154306:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536677.154308:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536677.154308:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536677.154310:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584536677.154310:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584536677.154312:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18476 00000100:00080000:18.0:1584536677.154314:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584536677.154321:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536677.154322:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536677.154323:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536677.154324:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536677.154325:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536677.154326:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536677.154326:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536677.154328:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536677.207618:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584536677.207622:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584536677.207624:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584536677.207624:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584536677.207626:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584536677.207628:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584536677.207629:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584536677.207631:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536757 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584536677.207640:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078426144 new x1661492078426224 00000800:00020000:2.0:1584536682.916063:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536682.916065:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536682.916067:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536682.916509:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584536682.928587:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536677/real 1584536677] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536764 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584536682.928595:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584536682.928596:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584536682.928597:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584536682.928603:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536682.928605:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536682.928606:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536682.928607:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536682.928607:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536682.928609:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584536682.928609:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584536682.928612:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18483 00000100:00080000:18.0:1584536682.928614:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584536682.928622:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536682.928624:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536682.928624:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536682.928626:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536682.928626:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536682.928627:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536682.928628:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536682.928629:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536682.984089:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584536682.984093:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584536682.984094:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584536682.984095:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584536682.984096:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584536682.984098:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584536682.984099:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584536682.984101:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536764 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584536682.984131:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078426224 new x1661492078426256 00000800:00020000:4.0:1584536688.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536688.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536688.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536688.916486:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584536688.928538:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536682/real 1584536682] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536769 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584536688.928546:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584536688.928547:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584536688.928549:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584536688.928554:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536688.928556:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536688.928557:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536688.928558:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536688.928559:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536688.928561:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584536688.928562:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584536688.928563:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18488 00000100:00080000:18.0:1584536688.928565:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584536688.928574:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536688.928576:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536688.928576:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536688.928577:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536688.928578:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536688.928579:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536688.928580:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536688.928581:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536688.981335:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584536688.981339:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584536688.981340:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584536688.981341:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584536688.981342:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584536688.981344:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584536688.981346:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584536688.981347:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536769 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584536688.981357:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078426256 new x1661492078426304 00000800:00020000:2.0:1584536694.916081:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536694.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536694.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536694.916517:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584536694.928571:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536688/real 1584536688] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536775 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584536694.928579:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584536694.928580:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584536694.928581:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584536694.928587:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536694.928588:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536694.928589:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536694.928591:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536694.928592:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536694.928593:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584536694.928594:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584536694.928595:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18494 00000100:00080000:18.0:1584536694.928597:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584536694.928606:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536694.928607:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536694.928608:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536694.928609:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536694.928609:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536694.928611:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536694.928611:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536694.928612:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536694.985751:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584536694.985755:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584536694.985756:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584536694.985757:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584536694.985758:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584536694.985760:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584536694.985762:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584536694.985764:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536775 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584536694.985772:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078426304 new x1661492078426336 00000800:00000100:2.0:1584536701.141564:0:4532:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536701.142068:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584536701.154113:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536694/real 1584536694] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536781 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584536701.154121:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584536701.154122:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584536701.154123:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584536701.154139:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536701.154141:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536701.154145:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536701.154146:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536701.154147:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536701.154148:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584536701.154149:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584536701.154151:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18500 00000100:00080000:18.0:1584536701.154152:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584536701.154158:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536701.154159:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536701.154160:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536701.154161:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536701.154162:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536701.154163:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584536701.154163:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584536701.154164:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536701.209426:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584536701.209430:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584536701.209432:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584536701.209433:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584536701.209434:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584536701.209436:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584536701.209437:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584536701.209439:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536781 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584536701.209468:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078426336 new x1661492078426400 00000800:00020000:2.0:1584536706.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536706.916085:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536706.916089:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536706.916526:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584536706.928665:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536701/real 1584536701] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536788 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584536706.928674:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584536706.928675:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584536706.928676:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584536706.928682:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536706.928684:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536706.928685:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536706.928686:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536706.928687:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536706.928688:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584536706.928689:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584536706.928691:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18507 00000100:00080000:20.0:1584536706.928693:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584536706.928703:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536706.928705:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536706.928705:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536706.928708:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536706.928708:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536706.928710:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536706.928710:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536706.928711:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536706.993680:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584536706.993685:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584536706.993687:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584536706.993687:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584536706.993689:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584536706.993691:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584536706.993692:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584536706.993694:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536788 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584536706.993705:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078426400 new x1661492078426448 00000800:00020000:4.0:1584536712.916061:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536712.916065:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536712.916068:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536712.916520:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584536712.928633:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536706/real 1584536706] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536793 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584536712.928644:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584536712.928645:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584536712.928647:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584536712.928656:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536712.928658:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536712.928659:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536712.928661:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536712.928662:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536712.928663:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584536712.928664:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584536712.928666:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18512 00000100:00080000:20.0:1584536712.928669:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584536712.928685:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536712.928687:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536712.928688:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536712.928689:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536712.928690:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536712.928691:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536712.928692:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536712.928693:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536712.977894:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584536712.977901:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584536712.977903:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584536712.977904:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584536712.977906:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584536712.977909:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584536712.977910:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584536712.977912:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536793 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584536712.977924:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078426448 new x1661492078426480 00000800:00020000:4.0:1584536718.916094:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536718.916101:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536718.916105:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536718.916524:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584536718.928621:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536712/real 1584536712] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536799 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584536718.928631:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584536718.928644:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584536718.928646:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584536718.928655:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536718.928657:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536718.928658:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536718.928659:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536718.928660:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536718.928661:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584536718.928662:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584536718.928665:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18518 00000100:00080000:20.0:1584536718.928669:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584536718.928684:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536718.928686:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536718.928687:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536718.928688:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536718.928688:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536718.928690:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536718.928690:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536718.928692:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536718.972456:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584536718.972463:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584536718.972465:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584536718.972466:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584536718.972468:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584536718.972470:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584536718.972472:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584536718.972474:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536799 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584536718.972522:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078426480 new x1661492078426528 00000800:00000100:0.0:1584536725.140702:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536725.141281:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584536725.153332:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536718/real 1584536718] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536805 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584536725.153340:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584536725.153342:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584536725.153343:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584536725.153362:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536725.153364:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536725.153373:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536725.153375:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536725.153375:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536725.153377:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584536725.153379:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584536725.153381:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18524 00000100:00080000:20.0:1584536725.153383:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584536725.153390:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536725.153391:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536725.153392:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536725.153393:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536725.153394:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536725.153395:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536725.153396:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536725.153397:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536725.215366:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584536725.215372:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584536725.215375:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584536725.215376:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584536725.215377:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584536725.215380:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584536725.215381:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584536725.215383:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536805 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584536725.215393:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078426528 new x1661492078426592 00000800:00020000:6.0:1584536730.916070:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584536730.916073:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584536730.916075:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536730.916508:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584536730.928593:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536725/real 1584536725] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536812 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584536730.928601:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584536730.928613:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584536730.928614:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584536730.928620:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536730.928621:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536730.928622:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536730.928623:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536730.928625:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536730.928626:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584536730.928627:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584536730.928629:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18531 00000100:00080000:20.0:1584536730.928631:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584536730.928640:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536730.928641:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536730.928642:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536730.928643:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536730.928644:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536730.928645:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536730.928646:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536730.928647:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536730.980259:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584536730.980263:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584536730.980265:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584536730.980266:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584536730.980267:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584536730.980269:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584536730.980271:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584536730.980272:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536812 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584536730.980301:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078426592 new x1661492078426624 00000800:00020000:2.0:1584536736.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536736.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536736.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536736.916466:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584536736.928518:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536730/real 1584536730] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536817 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584536736.928526:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584536736.928527:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584536736.928528:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584536736.928533:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536736.928535:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536736.928537:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536736.928538:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536736.928538:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536736.928540:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584536736.928540:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584536736.928542:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18536 00000100:00080000:20.0:1584536736.928544:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584536736.928554:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536736.928555:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536736.928556:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536736.928557:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536736.928557:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536736.928559:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536736.928559:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536736.928560:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536736.984540:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584536736.984544:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584536736.984546:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584536736.984546:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584536736.984548:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584536736.984550:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584536736.984551:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584536736.984553:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536817 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584536736.984587:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078426624 new x1661492078426672 00000800:00020000:0.0:1584536742.916056:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584536742.916058:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584536742.916060:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536742.916445:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584536742.928533:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536736/real 1584536736] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536823 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584536742.928553:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584536742.928554:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584536742.928555:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584536742.928561:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536742.928563:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536742.928564:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536742.928565:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536742.928566:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536742.928567:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584536742.928568:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584536742.928569:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18542 00000100:00080000:20.0:1584536742.928571:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584536742.928580:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536742.928581:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536742.928581:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536742.928583:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536742.928583:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536742.928584:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536742.928585:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536742.928586:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536742.975087:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584536742.975095:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584536742.975098:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584536742.975100:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584536742.975103:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584536742.975107:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584536742.975111:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584536742.975114:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536823 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584536742.975164:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078426672 new x1661492078426704 00000800:00000100:0.0:1584536749.141066:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536749.141518:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584536749.153567:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536742/real 1584536742] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536829 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584536749.153575:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584536749.153577:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584536749.153578:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584536749.153591:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536749.153593:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536749.153602:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536749.153603:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536749.153604:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536749.153605:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584536749.153606:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584536749.153608:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18548 00000100:00080000:20.0:1584536749.153610:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584536749.153615:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536749.153616:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536749.153617:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536749.153618:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536749.153619:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536749.153620:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536749.153621:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536749.153622:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536749.212425:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584536749.212429:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584536749.212431:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584536749.212431:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584536749.212433:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584536749.212435:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584536749.212436:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584536749.212438:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536829 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584536749.212468:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078426704 new x1661492078426784 00000800:00020000:2.0:1584536754.916092:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536754.916097:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536754.916102:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536754.916614:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584536754.928706:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536749/real 1584536749] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536836 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584536754.928725:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584536754.928726:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584536754.928727:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584536754.928733:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536754.928734:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536754.928735:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536754.928736:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536754.928737:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536754.928738:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584536754.928739:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584536754.928740:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18555 00000100:00080000:20.0:1584536754.928742:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584536754.928750:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536754.928752:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536754.928752:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536754.928753:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536754.928754:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536754.928755:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536754.928756:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536754.928757:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536754.972853:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584536754.972872:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584536754.972874:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584536754.972875:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584536754.972876:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584536754.972878:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584536754.972879:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584536754.972881:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536836 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584536754.972927:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078426784 new x1661492078426816 00000800:00020000:2.0:1584536760.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536760.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536760.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536760.916489:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584536760.928547:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536754/real 1584536754] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536841 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584536760.928555:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584536760.928556:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584536760.928558:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584536760.928564:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536760.928565:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536760.928566:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536760.928567:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536760.928568:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536760.928569:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584536760.928570:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584536760.928571:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18560 00000100:00080000:20.0:1584536760.928573:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584536760.928582:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536760.928583:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536760.928584:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536760.928585:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536760.928586:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536760.928587:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536760.928589:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536760.928590:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584536760.992278:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584536760.992282:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584536760.992283:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584536760.992284:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584536760.992285:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584536760.992287:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584536760.992289:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584536760.992290:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536841 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584536760.992321:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078426816 new x1661492078426848 00000800:00020000:2.0:1584536766.916104:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536766.916111:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536766.916115:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536766.916620:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584536766.928709:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536760/real 1584536760] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536847 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584536766.928731:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584536766.928732:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584536766.928734:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584536766.928742:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536766.928744:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536766.928745:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536766.928746:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536766.928747:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536766.928748:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584536766.928749:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584536766.928751:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18566 00000100:00080000:20.0:1584536766.928755:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584536766.928768:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536766.928770:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536766.928770:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536766.928772:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536766.928773:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536766.928775:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536766.928775:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536766.928788:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584536766.985322:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584536766.985329:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584536766.985331:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584536766.985332:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584536766.985334:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584536766.985336:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584536766.985338:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584536766.985340:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536847 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584536766.985369:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078426848 new x1661492078426896 00000800:00000100:0.0:1584536773.141161:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536773.141638:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584536773.153693:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536766/real 1584536766] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536853 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584536773.153703:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584536773.153704:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584536773.153706:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584536773.153728:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536773.153730:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536773.153734:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536773.153735:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536773.153736:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536773.153738:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584536773.153739:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584536773.153741:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18572 00000100:00080000:20.0:1584536773.153744:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584536773.153752:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536773.153753:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536773.153755:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536773.153757:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536773.153757:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536773.153759:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536773.153759:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536773.153760:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584536773.208121:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584536773.208128:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584536773.208130:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584536773.208131:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584536773.208133:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584536773.208136:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584536773.208137:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584536773.208139:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536853 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584536773.208168:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078426896 new x1661492078426960 00000800:00020000:6.0:1584536778.916143:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584536778.938963:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584536778.963139:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536778.963556:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584536778.975648:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536773/real 1584536773] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536860 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584536778.975668:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584536778.975669:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584536778.975670:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584536778.975677:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536778.975678:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536778.975679:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536778.975680:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536778.975681:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536778.975682:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584536778.975683:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584536778.975684:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18579 00000100:00080000:20.0:1584536778.975687:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584536778.975697:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536778.975698:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536778.975699:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536778.975701:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536778.975701:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536778.975702:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536778.975703:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536778.975704:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584536779.015258:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584536779.015263:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584536779.015264:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584536779.015265:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584536779.015267:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584536779.015269:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584536779.015270:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584536779.015272:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536860 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584536779.015306:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078426960 new x1661492078427008 00000800:00020000:2.0:1584536784.916058:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536784.916060:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536784.916062:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536784.916420:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584536784.928470:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536779/real 1584536779] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536866 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584536784.928478:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584536784.928479:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584536784.928480:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584536784.928486:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536784.928488:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536784.928489:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536784.928490:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536784.928490:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536784.928492:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584536784.928492:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584536784.928495:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18584 00000100:00080000:20.0:1584536784.928497:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584536784.928506:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536784.928507:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536784.928507:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536784.928509:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536784.928509:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536784.928510:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536784.928511:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536784.928512:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584536784.936587:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584536784.936591:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584536784.936592:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584536784.936593:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584536784.936595:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584536784.936597:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584536784.936598:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584536784.936600:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536866 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584536784.936631:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078427008 new x1661492078427040 00000800:00020000:4.0:1584536790.916125:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536790.916130:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536790.916135:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536790.916603:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584536790.928696:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536784/real 1584536784] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536871 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584536790.928715:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584536790.928716:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584536790.928717:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584536790.928723:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536790.928724:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536790.928725:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536790.928726:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536790.928727:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536790.928729:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584536790.928730:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584536790.928731:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18590 00000100:00080000:20.0:1584536790.928733:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584536790.928742:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536790.928743:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536790.928743:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536790.928745:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536790.928745:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536790.928746:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536790.928747:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536790.928748:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584536790.936828:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584536790.936832:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584536790.936833:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584536790.936834:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584536790.936835:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584536790.936837:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584536790.936838:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584536790.936840:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536871 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584536790.936873:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078427040 new x1661492078427072 00000800:00000100:0.0:1584536797.141416:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536797.141855:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584536797.153910:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536790/real 1584536790] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536877 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584536797.153921:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584536797.153923:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584536797.153925:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584536797.153948:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536797.153950:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536797.153953:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536797.153956:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536797.153956:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536797.153958:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584536797.153959:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584536797.153961:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18596 00000100:00080000:20.0:1584536797.153965:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584536797.153973:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536797.153974:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536797.153975:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536797.153976:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536797.153977:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536797.153978:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536797.153979:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536797.153980:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584536797.162178:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584536797.162185:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584536797.162187:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584536797.162188:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584536797.162190:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584536797.162192:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584536797.162194:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584536797.162196:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536877 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584536797.162225:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078427072 new x1661492078427152 00000800:00020000:4.0:1584536802.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536802.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536802.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536802.916515:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584536802.928599:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536797/real 1584536797] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536884 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584536802.928618:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584536802.928619:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584536802.928620:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584536802.928626:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536802.928628:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536802.928629:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536802.928630:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536802.928631:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536802.928632:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584536802.928633:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584536802.928634:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18603 00000100:00080000:20.0:1584536802.928636:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584536802.928645:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536802.928646:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536802.928646:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536802.928647:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536802.928648:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536802.928649:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536802.928650:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536802.928651:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584536802.936857:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584536802.936861:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584536802.936862:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584536802.936863:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584536802.936865:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584536802.936867:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584536802.936869:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584536802.936871:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536884 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584536802.936900:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078427152 new x1661492078427184 00000800:00020000:6.0:1584536808.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584536808.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584536808.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536808.916468:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584536808.928522:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536802/real 1584536802] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536889 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584536808.928530:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584536808.928532:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584536808.928534:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584536808.928540:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536808.928541:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536808.928542:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536808.928543:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536808.928544:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536808.928545:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584536808.928546:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584536808.928548:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18608 00000100:00080000:20.0:1584536808.928550:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584536808.928559:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536808.928561:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536808.928561:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536808.928562:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536808.928563:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536808.928564:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536808.928565:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536808.928566:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584536808.936381:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584536808.936385:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584536808.936387:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584536808.936388:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584536808.936389:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584536808.936391:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584536808.936392:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584536808.936394:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536889 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584536808.936423:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078427184 new x1661492078427232 00000800:00020000:2.0:1584536814.916081:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536814.916086:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536814.916091:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536814.916497:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584536814.928571:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536808/real 1584536808] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536895 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584536814.928594:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584536814.928595:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584536814.928596:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584536814.928602:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536814.928603:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536814.928604:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536814.928605:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536814.928606:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536814.928607:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584536814.928608:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584536814.928609:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18614 00000100:00080000:20.0:1584536814.928611:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584536814.928619:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536814.928621:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536814.928621:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536814.928622:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536814.928623:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536814.928624:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536814.928625:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536814.928626:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584536814.936634:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584536814.936642:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584536814.936645:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584536814.936647:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584536814.936650:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584536814.936654:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584536814.936658:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584536814.936662:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536895 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584536814.936720:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078427232 new x1661492078427264 00000800:00000100:0.0:1584536821.141785:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536821.142349:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584536821.154397:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536814/real 1584536814] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536901 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584536821.154405:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584536821.154406:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584536821.154407:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584536821.154422:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536821.154424:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536821.154433:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536821.154434:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536821.154435:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536821.154437:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584536821.154437:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584536821.154439:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18620 00000100:00080000:20.0:1584536821.154441:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584536821.154447:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536821.154448:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536821.154448:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536821.154450:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536821.154450:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536821.154451:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536821.154453:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536821.154454:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536821.162047:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536821.162051:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536821.162053:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536821.162053:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536821.162055:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536821.162057:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536821.162058:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536821.162060:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536901 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584536821.162090:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078427264 new x1661492078427328 00000800:00020000:2.0:1584536826.916083:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536826.916086:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536826.916088:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536826.916527:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584536826.928578:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536821/real 1584536821] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536908 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584536826.928586:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584536826.928587:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584536826.928589:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584536826.928594:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536826.928596:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536826.928596:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536826.928598:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536826.928598:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536826.928600:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584536826.928600:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584536826.928602:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18627 00000100:00080000:20.0:1584536826.928604:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584536826.928612:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536826.928614:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536826.928614:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536826.928616:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536826.928617:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536826.928618:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536826.928619:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536826.928620:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536826.936388:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536826.936392:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536826.936394:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536826.936395:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536826.936396:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536826.936398:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536826.936400:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536826.936401:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536908 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584536826.936430:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078427328 new x1661492078427376 00000800:00020000:2.0:1584536832.916062:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536832.916066:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536832.916068:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536832.916495:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584536832.928558:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536826/real 1584536826] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536913 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584536832.928568:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584536832.928569:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584536832.928570:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584536832.928577:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536832.928579:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536832.928580:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536832.928581:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536832.928582:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536832.928584:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584536832.928584:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584536832.928586:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18632 00000100:00080000:20.0:1584536832.928589:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584536832.928602:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536832.928603:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536832.928605:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536832.928607:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536832.928608:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536832.928609:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536832.928609:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536832.928610:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536832.936500:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536832.936506:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536832.936508:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536832.936509:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536832.936511:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536832.936513:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536832.936515:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536832.936517:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536913 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584536832.936526:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078427376 new x1661492078427408 00000800:00020000:4.0:1584536838.916040:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536838.916043:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536838.916045:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536838.916527:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584536838.928616:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536832/real 1584536832] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536919 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584536838.928637:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584536838.928639:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584536838.928641:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584536838.928649:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536838.928651:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536838.928652:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536838.928653:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536838.928654:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536838.928656:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584536838.928657:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584536838.928659:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18638 00000100:00080000:20.0:1584536838.928662:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584536838.928678:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536838.928679:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536838.928680:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536838.928682:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536838.928682:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536838.928684:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536838.928684:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536838.928686:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536838.936632:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536838.936639:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536838.936641:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536838.936642:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536838.936644:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536838.936647:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536838.936648:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536838.936650:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536919 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584536838.936680:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078427408 new x1661492078427456 00000800:00000100:0.0:1584536845.140580:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536845.141012:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584536845.153064:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536838/real 1584536838] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536925 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584536845.153073:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584536845.153074:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584536845.153076:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584536845.153094:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536845.153096:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536845.153104:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536845.153105:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536845.153106:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536845.153107:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584536845.153108:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584536845.153111:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18644 00000100:00080000:20.0:1584536845.153114:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584536845.153120:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536845.153122:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536845.153123:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536845.153124:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536845.153125:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536845.153126:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584536845.153127:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584536845.153128:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536845.161024:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536845.161047:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536845.161049:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536845.161050:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536845.161052:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536845.161054:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536845.161056:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536845.161058:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536925 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584536845.161067:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078427456 new x1661492078427520 00000800:00020000:4.0:1584536850.916052:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536850.916055:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536850.916057:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536850.916472:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584536850.928524:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536845/real 1584536845] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536932 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584536850.928532:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584536850.928533:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584536850.928534:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584536850.928540:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536850.928542:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536850.928543:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536850.928544:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536850.928545:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536850.928546:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584536850.928547:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584536850.928548:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18651 00000100:00080000:22.0:1584536850.928550:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584536850.928562:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536850.928563:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536850.928564:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536850.928565:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536850.928565:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536850.928567:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536850.928568:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536850.928569:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536850.936352:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536850.936357:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536850.936359:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536850.936359:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536850.936361:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536850.936363:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536850.936365:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536850.936367:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536932 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584536850.936376:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078427520 new x1661492078427552 00000800:00020000:2.0:1584536856.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536856.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (4): c: 6, oc: 0, rc: 8 00000800:00000100:2.0:1584536856.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536856.916490:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584536856.928544:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536850/real 1584536850] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536937 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584536856.928552:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584536856.928553:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584536856.928555:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584536856.928560:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536856.928562:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536856.928563:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536856.928564:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536856.928565:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536856.928566:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584536856.928567:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584536856.928569:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18656 00000100:00080000:22.0:1584536856.928571:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584536856.928579:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536856.928580:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536856.928581:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536856.928582:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536856.928584:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536856.928585:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536856.928586:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536856.928587:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536856.936243:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536856.936247:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536856.936249:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536856.936250:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536856.936252:0:25724:0:(client.c:2816:ptlrpc_resend_req()) @@@ going to resend req@ffff8d2babe41680 x1661492078427584/t0(0) o17->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 456/432 e 0 to 0 dl 1584536923 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000100:00080000:16.0:1584536856.936258:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536856.936260:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536856.936261:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536856.936263:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536937 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584536856.936270:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078427552 new x1661492078427632 00000800:00020000:4.0:1584536862.916089:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536862.916092:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536862.916093:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536862.916510:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584536862.928561:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536856/real 1584536856] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536943 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584536862.928568:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584536862.928570:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584536862.928571:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584536862.928576:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536862.928578:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536862.928578:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536862.928579:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536862.928580:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536862.928581:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584536862.928582:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584536862.928584:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18662 00000100:00080000:22.0:1584536862.928586:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584536862.928594:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536862.928595:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536862.928597:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536862.928598:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536862.928599:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536862.928600:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536862.928601:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536862.928602:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536862.936556:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536862.936560:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536862.936561:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536862.936562:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536862.936563:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536862.936565:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536862.936567:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536862.936569:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536943 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584536862.936577:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078427632 new x1661492078427664 00000800:00000100:0.0:1584536869.141506:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536869.141913:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584536869.153963:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536862/real 1584536862] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536949 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584536869.153971:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584536869.153972:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584536869.153973:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584536869.153987:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536869.153989:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536869.153990:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536869.153991:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536869.153992:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536869.153993:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584536869.153994:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584536869.153995:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18668 00000100:00080000:22.0:1584536869.153997:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584536869.154004:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536869.154005:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536869.154006:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536869.154007:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536869.154007:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536869.154009:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536869.154009:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536869.154010:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536869.161821:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536869.161825:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536869.161826:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536869.161827:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536869.161829:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536869.161831:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536869.161833:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536869.161834:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536949 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584536869.161863:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078427664 new x1661492078427728 00000800:00020000:2.0:1584536874.916035:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536874.916037:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536874.916039:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536874.916697:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584536874.928753:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536869/real 1584536869] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536956 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584536874.928761:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584536874.928762:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584536874.928764:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584536874.928769:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536874.928770:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536874.928778:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536874.928779:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536874.928780:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536874.928781:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584536874.928782:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584536874.928784:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18675 00000100:00080000:22.0:1584536874.928786:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584536874.928792:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536874.928793:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536874.928794:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536874.928795:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536874.928796:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536874.928797:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536874.928797:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536874.928799:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536874.937517:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536874.937521:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536874.937523:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536874.937523:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536874.937525:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536874.937526:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536874.937528:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536874.937530:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536956 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584536874.937538:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078427728 new x1661492078427776 00000800:00020000:2.0:1584536880.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536880.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536880.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536880.916489:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584536880.928541:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536874/real 1584536874] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536961 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584536880.928549:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584536880.928550:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584536880.928551:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584536880.928557:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536880.928559:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536880.928560:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536880.928561:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536880.928562:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536880.928564:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584536880.928565:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584536880.928567:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18680 00000100:00080000:22.0:1584536880.928569:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584536880.928578:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536880.928579:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536880.928580:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536880.928581:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536880.928581:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536880.928582:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536880.928583:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536880.928584:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536880.936680:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536880.936684:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536880.936686:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536880.936687:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536880.936688:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536880.936690:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536880.936691:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536880.936693:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536961 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584536880.936701:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078427776 new x1661492078427808 00000800:00020000:2.0:1584536886.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536886.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536886.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536886.916523:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584536886.928572:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536880/real 1584536880] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536967 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584536886.928580:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584536886.928581:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584536886.928582:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584536886.928588:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536886.928590:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536886.928590:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536886.928592:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536886.928593:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536886.928594:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584536886.928595:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584536886.928597:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18686 00000100:00080000:22.0:1584536886.928599:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584536886.928607:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536886.928609:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536886.928609:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536886.928610:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536886.928611:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536886.928612:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536886.928613:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536886.928614:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536886.936451:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536886.936455:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536886.936457:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536886.936458:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536886.936460:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536886.936462:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536886.936463:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536886.936465:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536967 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584536886.936474:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078427808 new x1661492078427856 00000800:00000100:0.0:1584536893.141188:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536893.141644:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584536893.153757:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536886/real 1584536886] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536973 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584536893.153767:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584536893.153769:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584536893.153770:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584536893.153791:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536893.153794:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536893.153795:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536893.153796:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536893.153797:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536893.153810:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584536893.153811:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584536893.153813:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18692 00000100:00080000:22.0:1584536893.153816:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584536893.153823:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536893.153825:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536893.153825:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536893.153827:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536893.153827:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536893.153829:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536893.153829:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536893.153830:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000800:00000100:0.0:1584536898.169077:0:164947:0:(o2iblnd_cb.c:3200:kiblnd_cm_callback()) 10.10.2.22@o2ib: ROUTE ERROR -110 00000800:00000100:0.0:1584536898.169081:0:164947:0:(o2iblnd_cb.c:2273:kiblnd_peer_connect_failed()) Deleting messages for 10.10.2.22@o2ib: connection failed 00000400:00000100:0.0:1584536898.169084:0:164947:0:(lib-msg.c:705:lnet_health_check()) msg 0@<0:0>->10.10.2.22@o2ib exceeded retry count 0 00000100:00080000:16.0:1584536898.169093:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:16.0:1584536898.169096:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0003_UUID on 10.10.2.22@o2ib failed (-110) 00000100:00080000:22.0:1584536918.169526:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536918.169529:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536918.169535:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536918.169536:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536918.169538:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:22.0:1584536918.169540:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584536918.169542:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584536918.169545:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18699 00000100:00080000:22.0:1584536918.169547:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 14562 00000100:00080000:22.0:1584536918.169550:0:4063:0:(import.c:596:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0003 (at 10.10.2.21@o2ib) 00000100:00080000:22.0:1584536918.169551:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.21@o2ib/10.10.2.21@o2ib 00000100:00080000:22.0:1584536918.169560:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536918.169561:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536918.169562:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536918.169564:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536918.169565:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536918.169566:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536918.169566:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536918.169568:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536918.169814:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:16.0:1584536918.169816:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0003_UUID on 10.10.2.21@o2ib failed (-19) 00000100:00000400:0.0:1584536923.377069:0:25730:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584536852/real 1584536852] req@ffff8d2babe45e80 x1661492078427600/t0(0) o17->public1-OST0001-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 456/432 e 0 to 1 dl 1584536923 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 00000100:02000400:0.0:1584536923.377079:0:25730:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection to public1-OST0001 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:0.0:1584536923.377080:0:25730:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from FULL to DISCONN 00000100:00080000:0.0:1584536923.377082:0:25730:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0001_UUID: waking up pinger s:DISCONN 00000100:00080000:0.0:1584536923.377089:0:166:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1584536923.377091:0:166:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1584536923.377091:0:166:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1584536923.377093:0:166:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:0.0:1584536923.377093:0:166:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1584536923.377095:0:166:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 17546 00000100:00080000:0.0:1584536923.377097:0:166:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:0.0:1584536923.377107:0:166:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1584536923.377108:0:166:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1584536923.377109:0:166:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1584536923.377110:0:166:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1584536923.377111:0:166:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1584536923.377112:0:166:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1584536923.377113:0:166:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1584536923.377114:0:166:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1584536923.377114:0:166:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1584536923.377115:0:166:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536923.377314:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536923.377321:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536923.377323:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0001_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536923.377324:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536923.377326:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536923.377329:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536923.377331:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536923.377333:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727d00): req@ffff8d2babe45e80 x1661492078427600/t0(0) o17->public1-OST0001-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 456/432 e 0 to 1 dl 1584536923 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 00000100:00080000:0.0:1584536948.441070:0:166:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1584536948.441073:0:166:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1584536948.441079:0:166:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1584536948.441082:0:166:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1584536948.441083:0:166:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:0.0:1584536948.441085:0:166:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:0.0:1584536948.441086:0:166:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1584536948.441090:0:166:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18699 00000100:00080000:0.0:1584536948.441092:0:166:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 18724 00000100:00080000:0.0:1584536948.441093:0:166:0:(import.c:570:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: tried all connections, increasing latency to 6s 00000100:00080000:0.0:1584536948.441096:0:166:0:(import.c:596:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0003 (at 10.10.2.22@o2ib) 00000100:00080000:0.0:1584536948.441097:0:166:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:0.0:1584536948.441111:0:166:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1584536948.441113:0:166:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1584536948.441114:0:166:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1584536948.441115:0:166:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1584536948.441116:0:166:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1584536948.441117:0:166:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1584536948.441118:0:166:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1584536948.441119:0:166:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536948.441256:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536948.441264:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536948.441266:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536948.441268:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536948.441270:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536948.441273:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536948.441274:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536948.441277:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584536973 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584536948.441289:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078427856 new x1661492078428080 00000800:00020000:16.0:1584536953.916194:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:16.0:1584536953.916197:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:16.0:1584536953.916199:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536953.916618:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584536953.928685:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536948/real 1584536948] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537035 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584536953.928695:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584536953.928697:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584536953.928699:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584536953.928706:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536953.928708:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536953.928709:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536953.928710:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536953.928711:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536953.928713:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584536953.928714:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584536953.928715:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18754 00000100:00080000:22.0:1584536953.928718:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584536953.928733:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536953.928734:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536953.928735:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536953.928736:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536953.928737:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536953.928738:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536953.928739:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536953.928740:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536953.948196:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536953.948202:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536953.948205:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536953.948206:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536953.948208:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536953.948210:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536953.948212:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536953.948214:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537035 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584536953.948243:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078428080 new x1661492078428112 00000800:00000100:0.0:1584536960.141026:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536960.141486:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584536960.153548:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536953/real 1584536953] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537040 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584536960.153557:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584536960.153559:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584536960.153561:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584536960.153567:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536960.153569:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536960.153569:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536960.153570:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536960.153571:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536960.153573:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584536960.153573:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584536960.153575:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18759 00000100:00080000:22.0:1584536960.153577:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584536960.153587:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536960.153589:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536960.153589:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536960.153591:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536960.153591:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536960.153592:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536960.153593:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536960.153594:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536960.200413:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536960.200419:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536960.200421:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536960.200422:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536960.200424:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536960.200426:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536960.200428:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536960.200430:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537040 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584536960.200438:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078428112 new x1661492078428160 00000800:00020000:2.0:1584536965.916073:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536965.916075:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536965.916077:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536965.916574:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584536965.928625:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536960/real 1584536960] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537047 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584536965.928634:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584536965.928635:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584536965.928636:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584536965.928641:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536965.928643:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536965.928644:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536965.928645:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536965.928646:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536965.928647:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584536965.928648:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584536965.928649:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18766 00000100:00080000:22.0:1584536965.928651:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584536965.928660:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536965.928661:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536965.928662:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536965.928663:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536965.928664:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536965.928665:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536965.928665:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536965.928667:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536965.965285:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536965.965289:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536965.965290:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536965.965291:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536965.965292:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536965.965295:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536965.965297:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536965.965298:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537047 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584536965.965328:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078428160 new x1661492078428192 00000800:00020000:4.0:1584536971.916127:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536971.916130:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536971.916132:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536971.916770:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584536971.928840:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536965/real 1584536965] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537052 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584536971.928848:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584536971.928849:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584536971.928850:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584536971.928865:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536971.928867:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536971.928872:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536971.928873:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536971.928874:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536971.928875:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584536971.928875:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584536971.928877:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18771 00000100:00080000:22.0:1584536971.928879:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584536971.928884:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536971.928886:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536971.928886:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536971.928887:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536971.928888:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536971.928889:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584536971.928891:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584536971.928892:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536971.965663:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536971.965672:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536971.965675:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536971.965677:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536971.965680:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536971.965684:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536971.965687:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536971.965691:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537052 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584536971.965712:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078428192 new x1661492078428272 00000800:00020000:6.0:1584536977.916142:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584536977.916144:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584536977.916146:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536977.916584:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584536977.928663:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536971/real 1584536971] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537058 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584536977.928671:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584536977.928672:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584536977.928673:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584536977.928680:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536977.928682:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536977.928682:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536977.928684:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536977.928684:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536977.928686:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584536977.928686:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584536977.928688:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18777 00000100:00080000:4.0:1584536977.928690:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584536977.928699:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536977.928700:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536977.928701:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536977.928702:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536977.928702:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536977.928704:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536977.928704:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536977.928705:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536977.968560:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536977.968564:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536977.968566:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536977.968567:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536977.968568:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536977.968570:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536977.968571:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536977.968573:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537058 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584536977.968602:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078428272 new x1661492078428304 00000800:00000100:0.0:1584536984.141178:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584536984.141616:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584536984.153693:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536977/real 1584536977] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537064 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584536984.153701:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584536984.153702:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584536984.153704:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584536984.153709:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536984.153711:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536984.153712:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536984.153713:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536984.153714:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536984.153715:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584536984.153715:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584536984.153717:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18783 00000100:00080000:4.0:1584536984.153719:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584536984.153727:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536984.153729:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536984.153729:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536984.153730:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536984.153731:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536984.153732:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536984.153733:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536984.153734:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536984.195285:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536984.195289:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536984.195291:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536984.195292:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536984.195293:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536984.195296:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536984.195297:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536984.195299:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537064 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584536984.195328:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078428304 new x1661492078428336 00000800:00020000:4.0:1584536989.916146:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584536989.916149:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584536989.916151:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536989.916596:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584536989.928666:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536984/real 1584536984] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537071 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584536989.928674:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584536989.928675:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584536989.928676:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584536989.928682:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536989.928683:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536989.928684:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536989.928685:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536989.928686:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536989.928687:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584536989.928688:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584536989.928689:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18790 00000100:00080000:4.0:1584536989.928691:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584536989.928700:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536989.928701:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536989.928702:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536989.928703:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536989.928704:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536989.928705:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536989.928706:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536989.928708:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536989.959474:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536989.959479:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536989.959480:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536989.959481:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536989.959482:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536989.959484:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536989.959486:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536989.959487:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537071 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584536989.959497:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078428336 new x1661492078428384 00000800:00020000:2.0:1584536995.916072:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584536995.916075:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584536995.916077:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584536995.916489:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584536995.928567:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536989/real 1584536989] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537076 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584536995.928575:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584536995.928577:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584536995.928578:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584536995.928595:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536995.928596:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536995.928602:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536995.928603:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536995.928604:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536995.928605:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584536995.928606:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584536995.928607:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18795 00000100:00080000:4.0:1584536995.928609:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584536995.928614:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536995.928615:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536995.928616:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536995.928617:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536995.928619:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536995.928620:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584536995.928620:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584536995.928621:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584536995.964807:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584536995.964812:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584536995.964813:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584536995.964814:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584536995.964815:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584536995.964817:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584536995.964819:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584536995.964820:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537076 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584536995.964829:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078428384 new x1661492078428448 00000800:00020000:2.0:1584537001.916060:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537001.916063:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537001.916064:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537001.916496:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537001.928633:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584536995/real 1584536995] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537082 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537001.928642:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537001.928644:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537001.928645:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537001.928652:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537001.928654:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537001.928654:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537001.928656:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537001.928656:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537001.928658:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537001.928658:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537001.928660:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18801 00000100:00080000:12.0:1584537001.928662:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537001.928674:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537001.928675:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537001.928676:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537001.928677:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537001.928678:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537001.928679:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537001.928679:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537001.928680:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537001.979307:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584537001.979313:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584537001.979315:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584537001.979316:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584537001.979318:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584537001.979322:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584537001.979323:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584537001.979325:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537082 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537001.979355:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078428448 new x1661492078428496 00000800:00000100:0.0:1584537008.141656:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537008.142090:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537008.154145:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537001/real 1584537001] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537088 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537008.154155:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537008.154157:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537008.154158:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537008.154166:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537008.154168:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537008.154169:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537008.154170:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537008.154171:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537008.154172:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537008.154173:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537008.154175:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18807 00000100:00080000:12.0:1584537008.154178:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537008.154191:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537008.154192:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537008.154193:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537008.154194:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537008.154195:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537008.154196:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537008.154197:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537008.154198:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537008.202891:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584537008.202896:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584537008.202899:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584537008.202900:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584537008.202901:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584537008.202903:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584537008.202905:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584537008.202907:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537088 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537008.202936:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078428496 new x1661492078428528 00000800:00020000:4.0:1584537013.916084:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537013.916087:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537013.916089:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537013.916565:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537013.928650:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537008/real 1584537008] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537095 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537013.928660:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537013.928661:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537013.928663:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537013.928671:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537013.928673:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537013.928674:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537013.928675:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537013.928676:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537013.928677:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537013.928678:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537013.928680:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18814 00000100:00080000:12.0:1584537013.928683:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537013.928698:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537013.928699:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537013.928700:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537013.928701:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537013.928702:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537013.928703:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537013.928705:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537013.928706:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537013.973768:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584537013.973775:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584537013.973777:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584537013.973778:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584537013.973780:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584537013.973782:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584537013.973784:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584537013.973786:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537095 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537013.973818:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078428528 new x1661492078428560 00000800:00020000:2.0:1584537019.916040:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537019.916043:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537019.916045:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537019.916533:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537019.928584:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537013/real 1584537013] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537100 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537019.928593:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537019.928595:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537019.928597:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537019.928617:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537019.928618:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537019.928624:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537019.928625:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537019.928626:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537019.928628:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537019.928629:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537019.928631:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18819 00000100:00080000:12.0:1584537019.928634:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537019.928641:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537019.928642:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537019.928643:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537019.928644:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537019.928646:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537019.928647:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537019.928648:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537019.928649:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537019.959640:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584537019.959646:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584537019.959649:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584537019.959650:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584537019.959652:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584537019.959656:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584537019.959657:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584537019.959659:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537100 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537019.959669:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078428560 new x1661492078428640 00000800:00020000:2.0:1584537025.916041:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537025.916044:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537025.916046:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537025.916492:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537025.928541:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537019/real 1584537019] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537106 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537025.928550:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537025.928552:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537025.928553:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537025.928560:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537025.928562:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537025.928563:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537025.928564:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537025.928565:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537025.928567:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537025.928567:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537025.928569:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18825 00000100:00080000:12.0:1584537025.928571:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537025.928583:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537025.928584:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537025.928586:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537025.928588:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537025.928588:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537025.928590:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537025.928590:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537025.928591:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537025.987004:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584537025.987011:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584537025.987013:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584537025.987014:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584537025.987015:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584537025.987018:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584537025.987019:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584537025.987021:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537106 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537025.987050:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078428640 new x1661492078428672 00000800:00000100:0.0:1584537032.138250:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537032.138710:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537032.150777:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537025/real 1584537025] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537112 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537032.150785:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537032.150786:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537032.150787:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537032.150793:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537032.150795:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537032.150795:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537032.150796:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537032.150797:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537032.150798:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537032.150799:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537032.150800:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18831 00000100:00080000:12.0:1584537032.150802:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537032.150813:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537032.150814:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537032.150814:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537032.150815:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537032.150816:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537032.150817:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537032.150817:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537032.150818:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537032.195386:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584537032.195395:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584537032.195399:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584537032.195401:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584537032.195404:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584537032.195408:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584537032.195412:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584537032.195416:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537112 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537032.195464:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078428672 new x1661492078428720 00000800:00020000:2.0:1584537037.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537037.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537037.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537037.916682:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537037.928725:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537032/real 1584537032] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537119 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537037.928732:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537037.928733:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537037.928735:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537037.928740:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537037.928742:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537037.928743:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537037.928744:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537037.928744:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537037.928746:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537037.928746:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537037.928749:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18838 00000100:00080000:12.0:1584537037.928751:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537037.928759:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537037.928761:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537037.928761:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537037.928763:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537037.928763:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537037.928764:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537037.928765:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537037.928766:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537037.974447:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584537037.974451:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584537037.974452:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584537037.974453:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584537037.974454:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584537037.974457:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584537037.974459:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584537037.974460:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537119 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537037.974492:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078428720 new x1661492078428752 00000800:00020000:2.0:1584537043.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537043.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537043.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537043.916455:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537043.928509:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537037/real 1584537037] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537124 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537043.928517:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537043.928518:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537043.928519:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537043.928534:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537043.928535:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537043.928541:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537043.928542:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537043.928542:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537043.928545:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537043.928545:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537043.928547:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18843 00000100:00080000:12.0:1584537043.928549:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537043.928554:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537043.928555:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537043.928556:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537043.928557:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537043.928558:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537043.928559:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537043.928560:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537043.928561:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537043.981848:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584537043.981852:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584537043.981853:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584537043.981854:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584537043.981855:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584537043.981857:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584537043.981858:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584537043.981860:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537124 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537043.981889:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078428752 new x1661492078428816 00000800:00020000:4.0:1584537049.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537049.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537049.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537049.916494:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537049.928548:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537043/real 1584537043] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537130 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537049.928556:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537049.928557:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537049.928558:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537049.928564:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537049.928566:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537049.928567:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537049.928569:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537049.928569:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537049.928571:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537049.928571:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537049.928573:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18849 00000100:00080000:12.0:1584537049.928575:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537049.928583:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537049.928584:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537049.928585:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537049.928586:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537049.928587:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537049.928588:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537049.928588:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537049.928590:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537049.977385:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584537049.977389:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584537049.977391:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584537049.977392:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584537049.977393:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584537049.977395:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584537049.977396:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584537049.977398:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537130 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537049.977427:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078428816 new x1661492078428864 00000800:00000100:0.0:1584537056.138597:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537056.139071:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537056.151122:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537049/real 1584537049] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537136 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537056.151129:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537056.151130:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537056.151131:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537056.151137:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537056.151139:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537056.151140:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537056.151141:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537056.151142:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537056.151143:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537056.151144:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537056.151146:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18855 00000100:00080000:12.0:1584537056.151147:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537056.151155:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537056.151157:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537056.151157:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537056.151158:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537056.151159:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537056.151160:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537056.151161:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537056.151162:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537056.219668:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584537056.219672:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584537056.219673:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584537056.219674:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584537056.219675:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584537056.219678:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584537056.219680:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584537056.219681:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537136 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537056.219710:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078428864 new x1661492078428896 00000800:00020000:2.0:1584537061.916073:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537061.916075:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537061.916077:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537061.916519:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537061.928587:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537056/real 1584537056] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537143 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537061.928595:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537061.928596:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537061.928598:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537061.928604:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537061.928606:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537061.928607:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537061.928608:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537061.928608:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537061.928610:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537061.928610:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537061.928612:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18862 00000100:00080000:12.0:1584537061.928614:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537061.928622:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537061.928624:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537061.928624:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537061.928625:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537061.928626:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537061.928627:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537061.928628:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537061.928629:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537061.965715:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584537061.965720:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584537061.965721:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584537061.965722:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584537061.965724:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584537061.965726:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584537061.965727:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584537061.965729:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537143 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537061.965757:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078428896 new x1661492078428944 00000800:00020000:0.0:1584537067.916086:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584537067.916089:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584537067.916092:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537067.916509:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537067.928601:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537061/real 1584537061] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537148 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537067.928614:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537067.928616:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537067.928617:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537067.928637:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537067.928639:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537067.928643:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537067.928645:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537067.928645:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537067.928647:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537067.928648:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537067.928650:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18867 00000100:00080000:12.0:1584537067.928653:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537067.928660:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537067.928661:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537067.928662:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537067.928663:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537067.928664:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537067.928665:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537067.928666:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537067.928667:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537067.998562:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537067.998568:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537067.998570:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537067.998571:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537067.998572:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537067.998575:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537067.998576:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537067.998578:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537148 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537067.998612:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078428944 new x1661492078429008 00000800:00020000:4.0:1584537073.916139:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537073.916143:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537073.916145:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537073.916624:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537073.928709:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537067/real 1584537067] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537154 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537073.928720:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537073.928721:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537073.928723:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537073.928730:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537073.928733:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537073.928733:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537073.928735:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537073.928735:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537073.928737:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537073.928738:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537073.928740:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18873 00000100:00080000:12.0:1584537073.928743:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537073.928760:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537073.928762:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537073.928763:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537073.928764:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537073.928764:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537073.928766:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537073.928767:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537073.928768:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537073.971979:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537073.971986:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537073.971988:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537073.971989:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537073.971991:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537073.971993:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537073.971995:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537073.971997:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537154 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537073.972006:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078429008 new x1661492078429040 00000800:00020000:0.0:1584537079.916149:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584537079.916152:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584537079.916154:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537079.916540:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537079.928595:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537073/real 1584537073] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537160 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537079.928604:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537079.928605:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537079.928607:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537079.928612:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537079.928614:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537079.928615:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537079.928616:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537079.928617:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537079.928619:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537079.928620:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537079.928621:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18879 00000100:00080000:12.0:1584537079.928624:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537079.928634:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537079.928635:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537079.928636:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537079.928637:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537079.928639:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537079.928640:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537079.928641:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537079.928642:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537079.984319:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537079.984324:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537079.984326:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537079.984328:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537079.984329:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537079.984331:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537079.984333:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537079.984335:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537160 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537079.984345:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078429040 new x1661492078429088 00000800:00020000:2.0:1584537085.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537085.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537085.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537085.916492:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537085.928544:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537079/real 1584537079] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537166 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537085.928552:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537085.928554:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537085.928555:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537085.928560:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537085.928562:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537085.928563:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537085.928564:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537085.928565:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537085.928566:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537085.928567:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537085.928569:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18885 00000100:00080000:12.0:1584537085.928571:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537085.928582:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537085.928583:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537085.928584:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537085.928585:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537085.928586:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537085.928587:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537085.928588:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537085.928589:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537085.970569:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537085.970575:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537085.970577:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537085.970578:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537085.970579:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537085.970582:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537085.970584:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537085.970585:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537166 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537085.970615:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078429088 new x1661492078429120 00000800:00020000:4.0:1584537091.916051:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537091.916053:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537091.916055:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537091.916526:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537091.928595:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537085/real 1584537085] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537172 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537091.928602:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537091.928603:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537091.928605:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537091.928619:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537091.928621:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537091.928626:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537091.928627:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537091.928628:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537091.928629:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537091.928630:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537091.928631:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18891 00000100:00080000:12.0:1584537091.928633:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537091.928639:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537091.928641:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537091.928641:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537091.928643:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537091.928643:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537091.928644:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537091.928645:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537091.928646:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537091.976566:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537091.976571:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537091.976573:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537091.976574:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537091.976575:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537091.976578:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537091.976579:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537091.976581:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537172 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537091.976611:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078429120 new x1661492078429200 00000800:00020000:2.0:1584537097.916063:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537097.916065:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537097.916067:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537097.916486:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537097.928538:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537091/real 1584537091] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537178 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537097.928546:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537097.928547:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537097.928548:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537097.928554:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537097.928556:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537097.928557:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537097.928558:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537097.928558:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537097.928560:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537097.928560:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537097.928563:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18897 00000100:00080000:12.0:1584537097.928565:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537097.928574:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537097.928576:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537097.928576:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537097.928578:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537097.928578:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537097.928579:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537097.928580:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537097.928581:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537097.977333:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537097.977337:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537097.977339:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537097.977340:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537097.977341:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537097.977343:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537097.977344:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537097.977346:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537178 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537097.977354:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078429200 new x1661492078429232 00000800:00020000:2.0:1584537103.916065:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537103.916068:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537103.916069:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537103.916557:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537103.928610:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537097/real 1584537097] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537184 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537103.928618:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537103.928619:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537103.928620:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537103.928625:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537103.928627:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537103.928628:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537103.928629:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537103.928630:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537103.928632:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537103.928633:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537103.928634:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18903 00000100:00080000:12.0:1584537103.928636:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537103.928644:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537103.928645:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537103.928646:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537103.928647:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537103.928648:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537103.928649:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537103.928650:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537103.928651:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537103.970740:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537103.970744:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537103.970745:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537103.970746:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537103.970747:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537103.970750:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537103.970751:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537103.970753:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537184 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537103.970761:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078429232 new x1661492078429264 00000800:00020000:6.0:1584537109.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584537109.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584537109.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537109.916624:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537109.928680:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537103/real 1584537103] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537190 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537109.928687:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537109.928688:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537109.928690:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537109.928695:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537109.928697:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537109.928698:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537109.928700:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537109.928701:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537109.928702:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537109.928703:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537109.928704:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18909 00000100:00080000:12.0:1584537109.928706:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537109.928714:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537109.928715:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537109.928716:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537109.928717:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537109.928718:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537109.928719:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537109.928719:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537109.928721:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537109.981862:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537109.981866:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537109.981867:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537109.981868:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537109.981870:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537109.981871:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537109.981873:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537109.981875:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537190 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537109.981903:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078429264 new x1661492078429312 00000800:00020000:2.0:1584537115.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537115.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537115.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537115.916490:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537115.928541:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537109/real 1584537109] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537196 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537115.928549:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537115.928550:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537115.928551:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537115.928563:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537115.928565:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537115.928570:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537115.928572:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537115.928572:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537115.928574:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537115.928574:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537115.928576:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18915 00000100:00080000:12.0:1584537115.928578:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537115.928583:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537115.928584:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537115.928585:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537115.928586:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537115.928587:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537115.928588:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537115.928589:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537115.928590:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537115.982184:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537115.982188:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537115.982189:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537115.982190:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537115.982191:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537115.982193:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537115.982194:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537115.982196:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537196 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584537115.982205:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078429312 new x1661492078429376 00000800:00020000:2.0:1584537121.916065:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537121.916067:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537121.916069:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537121.916511:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584537121.928652:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537115/real 1584537115] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537202 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584537121.928660:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584537121.928661:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584537121.928663:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584537121.928670:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537121.928672:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537121.928672:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537121.928674:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537121.928674:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537121.928677:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584537121.928677:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584537121.928679:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18921 00000100:00080000:16.0:1584537121.928681:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584537121.928693:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537121.928694:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537121.928695:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537121.928696:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537121.928696:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537121.928698:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537121.928698:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537121.928699:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537121.963837:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537121.963842:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537121.963845:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537121.963846:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537121.963847:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537121.963850:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537121.963852:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537121.963853:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537202 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584537121.963886:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078429376 new x1661492078429424 00000800:00020000:2.0:1584537127.916128:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537127.916131:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537127.916134:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537127.916645:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584537127.928702:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537121/real 1584537121] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537208 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584537127.928712:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584537127.928713:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584537127.928715:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584537127.928722:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537127.928724:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537127.928725:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537127.928728:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537127.928729:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537127.928730:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584537127.928731:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584537127.928733:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18927 00000100:00080000:16.0:1584537127.928736:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584537127.928750:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537127.928751:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537127.928752:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537127.928753:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537127.928754:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537127.928755:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537127.928755:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537127.928757:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537127.974908:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537127.974914:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537127.974916:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537127.974917:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537127.974919:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537127.974921:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537127.974922:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537127.974924:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537208 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584537127.974933:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078429424 new x1661492078429456 00000800:00020000:6.0:1584537133.916042:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584537133.916046:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584537133.916048:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537133.916471:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584537133.928526:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537127/real 1584537127] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537214 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584537133.928538:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584537133.928539:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584537133.928541:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584537133.928551:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537133.928554:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537133.928555:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537133.928556:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537133.928557:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537133.928559:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584537133.928560:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584537133.928562:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18933 00000100:00080000:16.0:1584537133.928566:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584537133.928585:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537133.928587:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537133.928587:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537133.928588:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537133.928589:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537133.928591:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537133.928591:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537133.928593:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537133.984273:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537133.984280:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537133.984283:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537133.984284:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537133.984286:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537133.984289:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537133.984290:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537133.984292:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537214 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584537133.984322:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078429456 new x1661492078429488 00000800:00020000:2.0:1584537139.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537139.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537139.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537139.916561:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584537139.928625:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537133/real 1584537133] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537220 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584537139.928634:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584537139.928635:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584537139.928638:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584537139.928654:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537139.928656:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537139.928661:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537139.928663:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537139.928663:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537139.928665:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584537139.928666:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584537139.928667:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18939 00000100:00080000:16.0:1584537139.928670:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584537139.928677:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537139.928678:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537139.928679:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537139.928680:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537139.928681:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537139.928682:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537139.928683:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537139.928684:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537139.978974:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537139.978980:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537139.978983:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537139.978984:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537139.978985:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537139.978989:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537139.978990:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537139.978992:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537220 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537139.979001:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078429488 new x1661492078429568 00000800:00020000:0.0:1584537145.916072:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584537145.916074:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584537145.916076:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537145.916476:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537145.928528:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537139/real 1584537139] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537226 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537145.928536:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537145.928537:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537145.928538:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537145.928545:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537145.928547:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537145.928547:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537145.928549:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537145.928549:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537145.928552:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537145.928552:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537145.928554:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18945 00000100:00080000:20.0:1584537145.928556:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537145.928565:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537145.928567:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537145.928567:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537145.928569:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537145.928569:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537145.928570:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537145.928571:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537145.928572:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537145.976386:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537145.976391:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537145.976392:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537145.976393:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537145.976394:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537145.976396:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537145.976397:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537145.976399:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537226 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537145.976407:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078429568 new x1661492078429600 00000800:00020000:4.0:1584537151.916178:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537151.916181:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537151.916183:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537151.916563:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537151.928648:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537145/real 1584537145] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537232 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537151.972423:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537151.972435:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537151.972436:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537151.972440:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537151.972442:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537151.972442:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537151.972444:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537151.972456:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537151.972457:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537151.972458:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537151.972460:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18951 00000100:00080000:20.0:1584537151.972462:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537151.972470:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537151.972471:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537151.972472:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537151.972473:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537151.972474:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537151.972475:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537151.972476:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537151.972477:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537152.019808:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537152.019816:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537152.019820:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537152.019822:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537152.019825:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537152.019829:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537152.019833:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537152.019837:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537232 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537152.019885:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078429600 new x1661492078429648 00000800:00020000:2.0:1584537157.916085:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537157.916087:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537157.916089:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537157.916523:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537157.928575:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537152/real 1584537152] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537239 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537157.928583:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537157.928584:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537157.928585:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537157.928590:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537157.928593:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537157.928594:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537157.928595:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537157.928596:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537157.928597:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537157.928598:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537157.928599:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18957 00000100:00080000:20.0:1584537157.928602:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537157.928610:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537157.928612:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537157.928612:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537157.928613:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537157.928614:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537157.928615:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537157.928616:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537157.928617:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537157.936396:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537157.936400:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537157.936401:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537157.936402:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537157.936404:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537157.936406:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537157.936408:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537157.936409:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537239 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537157.936418:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078429648 new x1661492078429680 00000800:00020000:2.0:1584537163.916086:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537163.916089:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537163.916090:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537163.916501:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537163.928556:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537157/real 1584537157] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537244 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537163.928564:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537163.928565:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537163.928567:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537163.928580:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537163.928582:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537163.928587:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537163.928588:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537163.928589:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537163.928590:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537163.928591:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537163.928592:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18963 00000100:00080000:20.0:1584537163.928594:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537163.928599:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537163.928601:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537163.928601:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537163.928602:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537163.928603:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537163.928604:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537163.928605:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537163.928606:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537163.936523:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537163.936527:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537163.936528:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537163.936529:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537163.936530:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537163.936532:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537163.936534:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537163.936536:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537244 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537163.936565:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078429680 new x1661492078429744 00000800:00020000:2.0:1584537169.916086:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537169.916088:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537169.916090:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537169.916590:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537169.928643:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537163/real 1584537163] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537250 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537169.928655:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537169.928657:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537169.928658:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537169.928663:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537169.928665:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537169.928665:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537169.928667:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537169.928667:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537169.928669:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537169.928669:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537169.928671:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18969 00000100:00080000:20.0:1584537169.928673:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537169.928682:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537169.928683:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537169.928684:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537169.928685:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537169.928685:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537169.928687:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537169.928687:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537169.928688:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537169.936803:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537169.936807:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537169.936809:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537169.936809:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537169.936811:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537169.936813:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537169.936814:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537169.936816:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537250 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537169.936845:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078429744 new x1661492078429792 00000800:00020000:0.0:1584537175.916066:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584537175.916068:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584537175.916070:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537175.916477:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537175.928526:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537169/real 1584537169] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537256 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537175.928535:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537175.928536:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537175.928537:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537175.928542:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537175.928544:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537175.928545:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537175.928546:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537175.928547:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537175.928548:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537175.928549:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537175.928550:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18975 00000100:00080000:20.0:1584537175.928552:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537175.928562:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537175.928563:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537175.928564:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537175.928565:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537175.928566:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537175.928567:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537175.928569:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537175.928570:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537175.936414:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537175.936418:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537175.936420:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537175.936421:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537175.936422:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537175.936425:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537175.936426:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537175.936428:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537256 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537175.936457:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078429792 new x1661492078429824 00000800:00000100:0.0:1584537182.141038:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537182.141634:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537182.153694:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537175/real 1584537175] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537262 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537182.153704:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537182.153705:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537182.153707:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537182.153714:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537182.153716:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537182.153717:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537182.153718:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537182.153719:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537182.153721:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537182.153722:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537182.153724:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18981 00000100:00080000:20.0:1584537182.153727:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537182.153739:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537182.153740:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537182.153741:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537182.153742:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537182.153744:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537182.153745:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537182.153746:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537182.153747:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537182.162051:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537182.162058:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537182.162060:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537182.162061:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537182.162063:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537182.162066:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537182.162067:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537182.162069:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537262 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537182.162078:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078429824 new x1661492078429872 00000800:00020000:2.0:1584537187.916082:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537187.916088:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537187.916093:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537187.916588:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537187.928681:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537182/real 1584537182] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537269 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537187.928691:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537187.928692:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537187.928694:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537187.928713:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537187.928715:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537187.928720:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537187.928721:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537187.928722:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537187.928724:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537187.928725:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537187.928727:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18988 00000100:00080000:20.0:1584537187.928729:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537187.928737:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537187.928738:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537187.928740:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537187.928742:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537187.928742:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537187.928744:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537187.928744:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537187.928746:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537187.936822:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537187.936829:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537187.936831:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537187.936832:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537187.936834:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537187.936837:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537187.936838:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537187.936840:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537269 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537187.936869:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078429872 new x1661492078429936 00000800:00020000:12.0:1584537193.916083:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:12.0:1584537193.916085:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:12.0:1584537193.916088:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537193.916501:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537193.928558:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537187/real 1584537187] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537274 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537193.928568:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537193.928569:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537193.928571:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537193.928577:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537193.928579:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537193.928579:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537193.928581:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537193.928581:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537193.928583:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537193.928584:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537193.928586:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18993 00000100:00080000:20.0:1584537193.928588:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537193.928603:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537193.928604:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537193.928605:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537193.928606:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537193.928607:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537193.928608:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537193.928609:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537193.928610:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537193.936448:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537193.936454:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537193.936457:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537193.936458:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537193.936460:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537193.936463:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537193.936465:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537193.936467:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537274 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537193.936496:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078429936 new x1661492078429968 00000800:00020000:0.0:1584537199.916180:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584537199.916183:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584537199.916185:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537199.916624:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537199.928668:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537193/real 1584537193] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537199.928676:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537199.928677:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537199.928679:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537199.928684:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537199.928686:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537199.928687:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537199.928689:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537199.928689:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537199.928691:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537199.928692:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537199.928694:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18999 00000100:00080000:20.0:1584537199.928697:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537199.928706:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537199.928707:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537199.928708:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537199.928710:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537199.928710:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537199.928711:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537199.928712:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537199.928713:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537199.936861:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537199.936866:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537199.936868:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537199.936869:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537199.936871:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537199.936873:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537199.936875:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537199.936876:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537199.936906:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078429968 new x1661492078430016 00000800:00000100:0.0:1584537206.141360:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537206.141788:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537206.153839:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537199/real 1584537199] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537206.153846:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537206.153848:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537206.153849:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537206.153854:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537206.153855:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537206.153856:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537206.153857:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537206.153858:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537206.153860:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537206.153861:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537206.153862:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19005 00000100:00080000:20.0:1584537206.153864:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537206.153873:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537206.153875:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537206.153875:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537206.153876:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537206.153877:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537206.153878:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537206.153879:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537206.153880:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537206.161867:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537206.161871:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537206.161873:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537206.161873:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537206.161875:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537206.161877:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537206.161878:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537206.161880:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537206.161888:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078430016 new x1661492078430048 00000800:00020000:2.0:1584537211.916063:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537211.916066:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537211.916068:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537211.916511:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537211.928578:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537206/real 1584537206] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537211.928586:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537211.928587:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537211.928588:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537211.928602:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537211.928603:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537211.928609:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537211.928611:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537211.928612:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537211.928613:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537211.928614:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537211.928615:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19012 00000100:00080000:20.0:1584537211.928617:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537211.928623:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537211.928624:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537211.928625:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537211.928626:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537211.928627:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537211.928628:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537211.928629:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537211.928630:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537211.936388:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584537211.936393:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584537211.936394:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584537211.936395:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584537211.936396:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584537211.936398:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584537211.936400:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584537211.936402:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537211.936427:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078430048 new x1661492078430128 00000800:00020000:2.0:1584537217.916071:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537217.916073:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537217.916075:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537217.916487:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537217.928571:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537211/real 1584537211] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537217.928579:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537217.928580:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537217.928581:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537217.928587:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537217.928590:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537217.928591:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537217.928592:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537217.928593:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537217.928594:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537217.928595:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537217.928596:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19017 00000100:00080000:20.0:1584537217.928598:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537217.928607:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537217.928608:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537217.928609:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537217.928610:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537217.928611:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537217.928612:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537217.928612:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537217.928614:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537217.936340:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584537217.936344:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584537217.936345:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584537217.936346:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584537217.936348:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584537217.936349:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584537217.936351:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584537217.936353:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537217.936382:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078430128 new x1661492078430160 00000800:00020000:2.0:1584537223.916071:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537223.916073:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537223.916075:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537223.916507:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537223.928561:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537217/real 1584537217] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537223.928569:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537223.953933:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537223.953936:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537223.953951:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537223.953952:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537223.953953:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537223.953954:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537223.953955:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537223.953956:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537223.953957:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537223.953958:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19023 00000100:00080000:20.0:1584537223.953960:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537223.953968:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537223.953969:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537223.953970:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537223.953971:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537223.953972:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537223.953973:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537223.953973:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537223.953975:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537223.961862:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584537223.961866:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584537223.961867:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584537223.961868:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584537223.961870:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584537223.961871:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584537223.961874:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584537223.979663:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:0.0:1584537223.979708:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078430160 new x1661492078430192 00000800:00000100:0.0:1584537230.141688:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537230.142145:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584537230.154213:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537223/real 1584537223] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584537230.154221:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584537230.154222:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584537230.154224:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584537230.154230:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584537230.154232:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537230.154233:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584537230.154234:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537230.154235:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584537230.154236:0:4532:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584537230.154237:0:4532:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584537230.154238:0:4532:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19029 00000100:00080000:2.0:1584537230.154241:0:4532:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584537230.154251:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584537230.154252:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537230.154253:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584537230.154254:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537230.154255:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584537230.154256:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537230.154256:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584537230.154258:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537230.162017:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584537230.162021:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584537230.162023:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584537230.162023:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584537230.162025:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584537230.162041:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584537230.162042:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584537230.162044:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584537230.162054:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078430192 new x1661492078430240 00000800:00020000:2.0:1584537235.916066:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537235.916069:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537235.916071:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537235.916543:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584537235.928621:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537230/real 1584537230] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537317 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584537235.928629:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584537235.928630:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584537235.928631:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584537235.928647:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537235.928650:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537235.928656:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537235.928657:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537235.928657:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537235.928659:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584537235.928660:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584537235.928661:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19036 00000100:00080000:4.0:1584537235.928663:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584537235.928668:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537235.928669:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537235.928670:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537235.928671:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537235.928672:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537235.928673:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537235.928674:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537235.928675:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537235.936492:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584537235.936496:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584537235.936497:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584537235.936498:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584537235.936499:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584537235.936501:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584537235.936503:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584537235.936505:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537317 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584537235.936514:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078430240 new x1661492078430304 00000800:00020000:2.0:1584537241.916060:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537241.916062:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537241.916064:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537241.916467:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584537241.928555:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537235/real 1584537235] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537322 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584537241.928563:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584537241.928564:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584537241.928566:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584537241.928572:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537241.928573:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537241.928574:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537241.928575:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537241.928576:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537241.928577:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584537241.928578:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584537241.928579:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19041 00000100:00080000:4.0:1584537241.928581:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584537241.928590:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537241.928591:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537241.928592:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537241.928593:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537241.928594:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537241.928595:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537241.928595:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537241.928596:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537241.936359:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584537241.936365:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584537241.936366:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584537241.936367:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584537241.936369:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584537241.936371:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584537241.936374:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584537241.936375:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537322 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584537241.936385:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078430304 new x1661492078430352 00000800:00020000:2.0:1584537247.916054:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537247.916057:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537247.916059:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537247.916483:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537247.928565:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537241/real 1584537241] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537328 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537247.928576:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537247.928577:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537247.928579:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537247.928586:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537247.928589:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537247.928589:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537247.928591:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537247.928591:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537247.928593:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537247.928594:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537247.928596:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19047 00000100:00080000:8.0:1584537247.928600:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537247.928617:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537247.928619:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537247.928619:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537247.928621:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537247.928621:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537247.928623:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537247.928623:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537247.928625:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537247.936523:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584537247.936530:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584537247.936532:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584537247.936533:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584537247.936535:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584537247.936538:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584537247.936539:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584537247.936541:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537328 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584537247.936570:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078430352 new x1661492078430384 00000800:00000100:0.0:1584537254.140121:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537254.140619:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537254.152691:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537247/real 1584537247] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537334 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537254.152702:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537254.152703:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537254.152705:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537254.152712:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537254.152714:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537254.152715:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537254.152717:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537254.152717:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537254.152719:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537254.152721:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537254.152723:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19053 00000100:00080000:8.0:1584537254.152726:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537254.152742:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537254.152744:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537254.152744:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537254.152746:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537254.152746:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537254.152747:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537254.152748:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537254.152749:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537254.160681:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584537254.160688:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584537254.160690:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584537254.160691:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584537254.160693:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584537254.160696:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584537254.160697:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584537254.160699:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537334 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584537254.160729:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078430384 new x1661492078430416 00000800:00020000:2.0:1584537259.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537259.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537259.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537259.916621:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537259.928700:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537254/real 1584537254] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537341 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537259.928709:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537259.928710:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537259.928711:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537259.928726:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537259.928728:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537259.928733:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537259.928735:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537259.928736:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537259.928738:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537259.928739:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537259.928741:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19060 00000100:00080000:8.0:1584537259.928743:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537259.928750:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537259.928751:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537259.928752:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537259.928753:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537259.928754:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537259.928755:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537259.928756:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537259.928757:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537259.936414:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584537259.936420:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584537259.936422:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584537259.936423:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584537259.936424:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584537259.936427:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584537259.936429:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584537259.936431:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537341 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584537259.936439:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078430416 new x1661492078430496 00000800:00020000:2.0:1584537265.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537265.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537265.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537265.916494:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584537265.928546:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537259/real 1584537259] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537346 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584537265.928553:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584537265.928555:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584537265.928556:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584537265.928562:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537265.928564:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537265.928564:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537265.928566:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537265.928566:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537265.928568:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584537265.928568:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584537265.928570:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19065 00000100:00080000:10.0:1584537265.928572:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584537265.928582:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537265.928583:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537265.928584:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537265.928585:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537265.928586:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537265.928587:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537265.928589:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537265.928590:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537265.936525:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584537265.936529:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584537265.936530:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584537265.936531:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584537265.936532:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584537265.936534:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584537265.936535:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584537265.936537:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537346 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584537265.936566:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078430496 new x1661492078430528 00000800:00020000:4.0:1584537271.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537271.916081:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537271.916083:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537271.916522:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584537271.928590:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537265/real 1584537265] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537352 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584537271.928597:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584537271.928598:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584537271.928600:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584537271.928605:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537271.928607:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537271.928607:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537271.928609:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537271.928609:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537271.928611:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584537271.928611:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584537271.928613:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19071 00000100:00080000:10.0:1584537271.928615:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584537271.928623:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537271.928624:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537271.928625:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537271.928626:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537271.928628:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537271.928629:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537271.928630:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537271.928631:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537271.936377:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584537271.936382:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584537271.936383:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584537271.936384:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584537271.936385:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584537271.936387:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584537271.936389:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584537271.936391:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537352 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584537271.936400:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078430528 new x1661492078430576 00000800:00000100:0.0:1584537278.140591:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537278.141056:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584537278.153105:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537271/real 1584537271] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537358 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584537278.153113:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584537278.153114:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584537278.153115:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584537278.153121:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537278.153123:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537278.153123:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537278.153125:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537278.153125:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537278.153127:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584537278.153127:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584537278.153129:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19077 00000100:00080000:10.0:1584537278.153131:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584537278.153139:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537278.153140:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537278.153142:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537278.153143:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537278.153144:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537278.153145:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537278.153145:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537278.153147:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537278.161259:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584537278.161263:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584537278.161264:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584537278.161265:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584537278.161266:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584537278.161268:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584537278.161271:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584537278.161272:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537358 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584537278.161301:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078430576 new x1661492078430608 00000800:00020000:2.0:1584537283.916115:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537283.916117:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537283.916119:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537283.916504:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584537283.928558:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537278/real 1584537278] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537365 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584537283.928566:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584537283.928567:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584537283.928568:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584537283.928582:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537283.928583:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537283.928589:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537283.928590:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537283.928591:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537283.928592:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584537283.928593:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584537283.928594:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19084 00000100:00080000:10.0:1584537283.928596:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584537283.928603:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537283.928604:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537283.928605:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537283.928606:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537283.928607:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537283.928608:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537283.928608:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537283.928610:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537283.936644:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537283.936648:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537283.936650:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537283.936651:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537283.936652:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537283.936654:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537283.936655:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537283.936657:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537365 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537283.936687:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078430608 new x1661492078430672 00000800:00020000:4.0:1584537289.916052:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537289.916055:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537289.916057:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537289.916506:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537289.928559:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537283/real 1584537283] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537370 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537289.928571:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537289.928572:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537289.928574:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537289.928579:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537289.928580:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537289.928581:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537289.928582:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537289.928583:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537289.928584:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537289.928585:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537289.928587:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19089 00000100:00080000:12.0:1584537289.928589:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537289.928597:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537289.928598:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537289.928599:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537289.928600:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537289.928601:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537289.928602:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537289.928603:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537289.928604:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537289.936374:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537289.936378:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537289.936379:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537289.936380:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537289.936382:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537289.936384:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537289.936386:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537289.936388:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537370 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537289.936420:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078430672 new x1661492078430720 00000800:00020000:2.0:1584537295.916052:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537295.916054:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537295.916056:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537295.916470:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537295.928524:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537289/real 1584537289] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537376 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537295.928532:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537295.928533:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537295.928534:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537295.928540:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537295.928542:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537295.928542:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537295.928543:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537295.928544:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537295.928545:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537295.928546:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537295.928548:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19095 00000100:00080000:12.0:1584537295.928550:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537295.928558:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537295.928559:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537295.928559:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537295.928561:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537295.928561:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537295.928562:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537295.928564:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537295.928565:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537295.936675:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537295.936679:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537295.936681:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537295.936682:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537295.936683:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537295.936685:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537295.936686:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537295.936688:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537376 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537295.936717:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078430720 new x1661492078430752 00000800:00000100:0.0:1584537302.140878:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537302.141365:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537302.153448:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537295/real 1584537295] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537382 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537302.153456:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537302.153457:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537302.153459:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537302.153465:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537302.153467:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537302.153467:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537302.153468:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537302.153469:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537302.153470:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537302.153471:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537302.153473:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19101 00000100:00080000:12.0:1584537302.153475:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537302.153484:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537302.153485:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537302.153485:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537302.153486:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537302.153488:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537302.153489:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537302.153501:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537302.153502:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537302.161768:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537302.161774:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537302.161776:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537302.161777:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537302.161778:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537302.161781:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537302.161782:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537302.161784:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537382 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537302.161816:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078430752 new x1661492078430800 00000800:00020000:4.0:1584537307.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537307.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537307.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537307.916491:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537307.928543:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537302/real 1584537302] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537389 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537307.928551:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537307.928552:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537307.928553:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537307.928567:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537307.928569:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537307.928575:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537307.928576:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537307.928577:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537307.928578:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537307.928579:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537307.928581:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19108 00000100:00080000:12.0:1584537307.928583:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537307.928588:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537307.928590:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537307.928591:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537307.928592:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537307.928593:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537307.928594:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537307.928595:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537307.928596:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537307.936925:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537307.936930:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537307.936932:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537307.936933:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537307.936935:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537307.936936:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537307.936938:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537307.936940:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537389 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537307.936968:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078430800 new x1661492078430864 00000800:00020000:4.0:1584537313.916063:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537313.916066:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537313.916068:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537313.916481:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537313.928545:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537307/real 1584537307] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537394 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537313.928554:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537313.928555:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537313.928557:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537313.928563:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537313.928565:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537313.928566:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537313.928567:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537313.928568:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537313.928570:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537313.928570:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537313.928572:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19113 00000100:00080000:12.0:1584537313.928575:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537313.928588:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537313.928589:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537313.928590:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537313.928591:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537313.928592:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537313.928593:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537313.928594:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537313.928595:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537313.979047:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537313.979052:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537313.979054:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537313.979056:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537313.979057:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537313.979059:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537313.979061:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537313.979063:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537394 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537313.979091:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078430864 new x1661492078430896 00000800:00020000:0.0:1584537319.916062:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584537319.916064:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584537319.916067:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537319.916483:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537319.928535:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537313/real 1584537313] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537400 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537319.928544:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537319.928545:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537319.928547:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537319.928553:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537319.928554:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537319.928555:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537319.928557:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537319.928557:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537319.928559:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537319.928560:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537319.928562:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19119 00000100:00080000:12.0:1584537319.928565:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537319.928575:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537319.928577:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537319.928577:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537319.928579:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537319.928579:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537319.928581:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537319.928581:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537319.928582:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537319.973985:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537319.973991:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537319.973993:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537319.973994:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537319.973995:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537319.973997:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537319.973999:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537319.974001:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537400 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537319.974047:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078430896 new x1661492078430944 00000800:00000100:0.0:1584537326.141110:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537326.141643:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537326.153693:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537319/real 1584537319] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537406 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537326.153701:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537326.153702:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537326.153703:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537326.153708:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537326.153710:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537326.153710:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537326.153711:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537326.153712:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537326.153714:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537326.153715:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537326.153716:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19125 00000100:00080000:12.0:1584537326.153718:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537326.153726:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537326.153728:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537326.153728:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537326.153729:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537326.153730:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537326.153731:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537326.153732:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537326.153733:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537326.198888:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537326.198892:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537326.198894:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537326.198895:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537326.198897:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537326.198899:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537326.198900:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537326.198902:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537406 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537326.198931:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078430944 new x1661492078430976 00000800:00020000:4.0:1584537331.916056:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537331.916059:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537331.916061:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537331.916474:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537331.928537:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537326/real 1584537326] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537413 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537331.928545:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537331.928546:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537331.928547:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537331.928560:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537331.928562:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537331.928567:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537331.928569:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537331.928570:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537331.928571:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537331.928572:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537331.928573:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19132 00000100:00080000:12.0:1584537331.928575:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537331.928580:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537331.928581:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537331.928582:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537331.928583:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537331.928583:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537331.928585:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537331.928585:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537331.928586:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537331.966776:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537331.966784:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537331.966788:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537331.966790:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537331.966792:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537331.966797:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537331.966800:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537331.966804:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537413 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537331.966851:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078430976 new x1661492078431056 00000800:00020000:4.0:1584537337.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537337.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537337.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537337.916522:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537337.928572:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537331/real 1584537331] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537418 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537337.928580:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537337.928581:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537337.928583:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537337.928588:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537337.928590:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537337.928590:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537337.928592:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537337.928592:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537337.928594:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537337.928594:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537337.928596:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19137 00000100:00080000:12.0:1584537337.928598:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537337.928606:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537337.928607:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537337.928608:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537337.928609:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537337.928610:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537337.928611:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537337.928612:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537337.928613:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537337.973536:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537337.973540:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537337.973541:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537337.973542:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537337.973543:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537337.973545:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537337.973546:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537337.973548:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537418 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537337.973577:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078431056 new x1661492078431088 00000800:00020000:2.0:1584537343.916105:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537343.916108:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537343.916110:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537343.916518:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537343.928570:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537337/real 1584537337] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537424 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537343.928578:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537343.928579:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537343.928581:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537343.928587:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537343.928588:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537343.928589:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537343.928590:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537343.928591:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537343.928592:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537343.928593:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537343.928594:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19143 00000100:00080000:12.0:1584537343.928596:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537343.928604:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537343.928606:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537343.928606:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537343.928607:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537343.928608:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537343.928609:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537343.928610:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537343.928611:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537343.969110:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537343.969114:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537343.969115:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537343.969116:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537343.969117:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537343.969119:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537343.969120:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537343.969122:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537424 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537343.969151:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078431088 new x1661492078431120 00000800:00000100:0.0:1584537350.141580:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537350.142131:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537350.154181:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537343/real 1584537343] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537430 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537350.154193:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537350.154194:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537350.154195:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537350.154200:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537350.154202:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537350.154203:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537350.154204:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537350.154205:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537350.154206:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537350.154207:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537350.154208:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19149 00000100:00080000:12.0:1584537350.154210:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537350.154219:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537350.154220:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537350.154221:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537350.154222:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537350.154223:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537350.154224:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537350.154224:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537350.154225:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537350.199422:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537350.199426:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537350.199428:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537350.199429:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537350.199430:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537350.199432:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537350.199433:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537350.199435:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537430 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537350.199445:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078431120 new x1661492078431168 00000800:00020000:4.0:1584537355.916090:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537355.916093:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537355.916095:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537355.916480:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537355.928556:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537350/real 1584537350] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537437 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537355.928564:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537355.928565:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537355.928566:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537355.928580:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537355.928582:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537355.928588:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537355.928589:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537355.928590:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537355.928591:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537355.928592:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537355.928593:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19156 00000100:00080000:12.0:1584537355.928595:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537355.928600:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537355.928602:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537355.928602:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537355.928603:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537355.928604:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537355.928605:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537355.928607:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537355.928608:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537355.975269:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537355.975273:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537355.975274:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537355.975275:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537355.975277:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537355.975278:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537355.975280:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537355.975282:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537437 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537355.975292:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078431168 new x1661492078431232 00000800:00020000:2.0:1584537361.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537361.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537361.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537361.916515:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537361.928572:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537355/real 1584537355] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537442 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537361.928580:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537361.928581:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537361.928582:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537361.928587:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537361.928589:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537361.928590:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537361.928591:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537361.928591:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537361.928593:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537361.928593:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537361.928595:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19161 00000100:00080000:12.0:1584537361.928597:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537361.928605:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537361.928606:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537361.928607:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537361.928608:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537361.928609:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537361.928610:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537361.928611:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537361.928612:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537361.974791:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537361.974795:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537361.974797:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537361.974798:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537361.974799:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537361.974801:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537361.974802:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537361.974804:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537442 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537361.974837:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078431232 new x1661492078431280 00000800:00020000:0.0:1584537367.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584537367.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584537367.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537367.916464:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537367.928517:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537361/real 1584537361] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537448 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537367.928526:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537367.928527:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537367.928529:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537367.928537:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537367.928539:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537367.928540:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537367.928541:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537367.928542:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537367.928543:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537367.928544:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537367.928546:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19167 00000100:00080000:12.0:1584537367.928549:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537367.928563:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537367.928564:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537367.928566:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537367.928567:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537367.928568:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537367.928569:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537367.928570:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537367.928571:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537367.977824:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537367.977830:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537367.977832:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537367.977833:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537367.977835:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537367.977837:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537367.977838:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537367.977840:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537448 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537367.977869:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078431280 new x1661492078431312 00000800:00000100:0.0:1584537374.142145:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537374.142591:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537374.154642:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537367/real 1584537367] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537454 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537374.154650:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537374.154652:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537374.154653:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537374.154659:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537374.154661:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537374.154661:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537374.154663:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537374.154663:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537374.154665:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537374.154665:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537374.154667:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19173 00000100:00080000:12.0:1584537374.154669:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537374.154683:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537374.154685:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537374.154686:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537374.154687:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537374.154687:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537374.154689:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537374.154689:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537374.154690:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537374.203039:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537374.203044:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537374.203046:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537374.203046:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537374.203048:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537374.203051:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537374.203053:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537374.203055:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537454 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537374.203085:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078431312 new x1661492078431344 00000800:00020000:0.0:1584537379.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584537379.938891:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584537379.963069:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537379.963481:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537379.975554:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537374/real 1584537374] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537461 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537379.975563:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537379.975564:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537379.975566:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537379.975583:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537379.975585:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537379.975591:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537379.975592:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537379.975593:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537379.975594:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537379.975595:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537379.975598:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19180 00000100:00080000:12.0:1584537379.975601:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537379.975608:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537379.975609:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537379.975610:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537379.975611:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537379.975612:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537379.975613:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537379.975614:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537379.975615:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537380.015690:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537380.015696:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537380.015697:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537380.015699:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537380.015700:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537380.015702:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537380.015704:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537380.015706:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537461 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537380.015735:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078431344 new x1661492078431424 00000800:00020000:0.0:1584537385.916073:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584537385.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584537385.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537385.916491:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537385.928545:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537380/real 1584537380] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537467 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537385.928553:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537385.928554:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537385.928555:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537385.928561:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537385.928562:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537385.928563:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537385.928564:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537385.928565:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537385.928567:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537385.928568:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537385.928569:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19185 00000100:00080000:12.0:1584537385.928571:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537385.928580:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537385.928581:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537385.928582:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537385.928584:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537385.928584:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537385.928585:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537385.928586:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537385.928587:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537385.977167:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584537385.977172:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584537385.977173:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584537385.977174:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584537385.977176:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584537385.977178:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584537385.977179:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584537385.977181:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537467 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537385.977211:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078431424 new x1661492078431456 00000800:00020000:4.0:1584537391.916133:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537391.916136:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537391.916138:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537391.916576:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537391.928660:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537385/real 1584537385] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537472 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537391.928668:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537391.928669:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537391.928670:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537391.928676:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537391.928678:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537391.928679:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537391.928681:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537391.928682:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537391.928683:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537391.928683:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537391.928685:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19191 00000100:00080000:12.0:1584537391.928687:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537391.928695:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537391.928697:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537391.928697:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537391.928698:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537391.928699:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537391.928700:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537391.928701:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537391.928702:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537391.990023:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584537391.990034:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584537391.990035:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584537391.990036:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584537391.990037:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584537391.990039:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584537391.990041:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584537391.990043:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537472 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537391.990052:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078431456 new x1661492078431504 00000800:00000100:0.0:1584537398.140151:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537398.140540:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537398.152590:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537391/real 1584537391] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537478 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537398.152597:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537398.152599:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537398.152600:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537398.152605:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537398.152607:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537398.152608:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537398.152609:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537398.152609:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537398.152611:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537398.152612:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537398.152613:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19197 00000100:00080000:12.0:1584537398.152615:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537398.152623:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537398.152625:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537398.152625:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537398.152626:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537398.152627:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537398.152628:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537398.152629:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537398.152630:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537398.205580:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584537398.205584:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584537398.205585:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584537398.205586:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584537398.205588:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584537398.205589:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584537398.205591:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584537398.205593:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537478 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584537398.205602:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078431504 new x1661492078431536 00000800:00020000:2.0:1584537403.916174:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537403.916177:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537403.916179:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537403.916592:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584537403.928655:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537398/real 1584537398] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537485 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584537403.928663:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584537403.928664:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584537403.928666:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584537403.928678:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537403.928680:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537403.928685:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537403.928686:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537403.928686:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537403.928688:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584537403.928688:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584537403.928690:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19204 00000100:00080000:12.0:1584537403.928692:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584537403.928697:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537403.928698:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537403.928699:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537403.928700:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537403.928701:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537403.928702:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537403.928703:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584537403.928704:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537403.981127:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584537403.981131:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584537403.981133:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584537403.981134:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584537403.981135:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584537403.981137:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584537403.981138:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584537403.981140:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537485 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584537403.981149:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078431536 new x1661492078431600 00000800:00020000:6.0:1584537409.916201:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584537409.916204:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584537409.916206:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537409.916741:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584537409.928817:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537403/real 1584537403] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537490 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584537409.928825:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584537409.928826:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584537409.928828:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584537409.928834:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537409.928836:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537409.928836:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537409.928838:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537409.928838:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537409.928840:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584537409.928840:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584537409.928842:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19209 00000100:00080000:18.0:1584537409.928844:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584537409.928854:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537409.928855:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537409.928856:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537409.928857:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537409.928858:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537409.928859:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537409.928859:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537409.928860:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537409.990954:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584537409.990958:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584537409.990960:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584537409.990960:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584537409.990962:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584537409.990964:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584537409.990965:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584537409.990967:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537490 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584537409.990996:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078431600 new x1661492078431648 00000800:00020000:2.0:1584537415.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537415.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537415.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537415.916490:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584537415.928575:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537409/real 1584537409] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537496 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584537415.928583:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584537415.928584:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584537415.928586:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584537415.928592:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537415.928594:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537415.928595:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537415.928596:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537415.928597:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537415.928598:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584537415.928599:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584537415.928600:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19215 00000100:00080000:18.0:1584537415.928602:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584537415.928611:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537415.928612:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537415.928613:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537415.928614:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537415.928615:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537415.928616:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537415.928616:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537415.928618:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537415.984787:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584537415.984791:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584537415.984792:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584537415.984793:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584537415.984794:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584537415.984796:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584537415.984798:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584537415.984799:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537496 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584537415.984829:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078431648 new x1661492078431680 00000800:00000100:0.0:1584537422.140791:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537422.141231:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584537422.153310:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537415/real 1584537415] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537502 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584537422.153329:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584537422.153330:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584537422.153332:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584537422.153338:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537422.153339:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537422.153340:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537422.153341:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537422.153342:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537422.153343:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584537422.153344:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584537422.153346:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19221 00000100:00080000:18.0:1584537422.153348:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584537422.153357:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537422.153358:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537422.153358:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537422.153360:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537422.153360:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537422.153361:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537422.153363:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537422.153376:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537422.217699:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584537422.217704:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584537422.217705:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584537422.217706:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584537422.217708:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584537422.217709:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584537422.217711:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584537422.217713:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537502 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584537422.217743:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078431680 new x1661492078431728 00000800:00020000:6.0:1584537427.916193:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584537427.916196:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584537427.916198:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537427.916668:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584537427.928747:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537422/real 1584537422] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537509 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584537427.928755:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584537427.928756:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584537427.928757:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584537427.928772:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537427.928774:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537427.928779:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537427.928780:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537427.928781:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537427.928783:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584537427.928783:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584537427.928785:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19228 00000100:00080000:18.0:1584537427.928787:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584537427.928792:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537427.928794:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537427.928794:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537427.928796:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537427.928797:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537427.928799:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537427.928799:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584537427.928801:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537427.981257:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584537427.981261:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584537427.981263:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584537427.981264:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584537427.981265:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584537427.981267:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584537427.981268:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584537427.981270:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537509 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537427.981299:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078431728 new x1661492078431792 00000800:00020000:2.0:1584537433.916051:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537433.916054:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537433.916056:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537433.916505:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537433.928559:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537427/real 1584537427] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537514 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537433.928574:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537433.928575:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537433.928576:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537433.928583:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537433.928585:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537433.928586:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537433.928588:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537433.928588:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537433.928590:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537433.928591:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537433.928593:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19233 00000100:00080000:20.0:1584537433.928595:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537433.928609:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537433.928610:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537433.928611:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537433.928612:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537433.928613:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537433.928614:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537433.928615:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537433.928616:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537433.988145:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584537433.988150:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584537433.988152:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584537433.988153:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584537433.988154:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584537433.988156:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584537433.988158:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584537433.988160:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537514 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537433.988188:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078431792 new x1661492078431824 00000800:00020000:0.0:1584537439.916097:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584537439.916100:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584537439.916102:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537439.916514:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537439.928558:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537433/real 1584537433] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537520 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537439.928568:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537439.928570:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537439.928572:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537439.928579:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537439.928581:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537439.928582:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537439.928583:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537439.928584:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537439.928586:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537439.928586:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537439.928589:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19239 00000100:00080000:20.0:1584537439.928592:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537439.928605:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537439.928606:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537439.928607:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537439.928608:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537439.928609:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537439.928610:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537439.928612:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537439.928613:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537439.974009:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584537439.974015:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584537439.974017:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584537439.974019:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584537439.974020:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584537439.974023:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584537439.974024:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584537439.974026:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537520 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537439.974073:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078431824 new x1661492078431872 00000800:00000100:0.0:1584537446.141264:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537446.141667:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537446.153720:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537439/real 1584537439] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537526 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537446.153729:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537446.153730:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537446.153732:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537446.153738:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537446.153740:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537446.153741:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537446.153742:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537446.153743:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537446.153744:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537446.153745:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537446.153747:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19245 00000100:00080000:20.0:1584537446.153749:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537446.153761:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537446.153763:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537446.153764:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537446.153765:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537446.153767:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537446.153768:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537446.153769:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537446.153770:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537446.208589:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584537446.208595:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584537446.208597:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584537446.208598:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584537446.208599:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584537446.208603:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584537446.208604:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584537446.208606:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537526 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537446.208635:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078431872 new x1661492078431904 00000800:00020000:4.0:1584537451.916180:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537451.916183:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537451.916185:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537451.916629:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537451.928700:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537446/real 1584537446] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537533 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537451.928708:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537451.928709:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537451.928710:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537451.928725:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537451.928727:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537451.928732:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537451.928733:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537451.928734:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537451.928736:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537451.928736:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537451.928738:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19252 00000100:00080000:20.0:1584537451.928740:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537451.928745:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537451.928747:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537451.928748:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537451.928750:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537451.928750:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537451.928751:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537451.928752:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537451.928753:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537451.991490:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584537451.991495:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584537451.991497:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584537451.991498:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584537451.991499:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584537451.991501:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584537451.991503:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584537451.991505:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537533 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537451.991534:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078431904 new x1661492078431984 00000800:00020000:2.0:1584537457.916069:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537457.916071:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537457.916073:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537457.916547:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537457.928600:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537451/real 1584537451] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537538 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537457.928608:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537457.928609:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537457.928610:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537457.928616:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537457.928618:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537457.928619:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537457.928620:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537457.928620:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537457.928622:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537457.928622:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537457.928624:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19257 00000100:00080000:20.0:1584537457.928626:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537457.928636:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537457.928638:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537457.928638:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537457.928640:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537457.928640:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537457.928641:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537457.928642:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537457.928643:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537457.986413:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584537457.986417:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584537457.986419:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584537457.986420:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584537457.986421:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584537457.986423:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584537457.986425:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584537457.986426:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537538 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537457.986456:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078431984 new x1661492078432016 00000800:00020000:4.0:1584537463.916124:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537463.916127:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537463.916129:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537463.916568:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537463.928622:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537457/real 1584537457] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537544 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537463.928630:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537463.928631:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537463.928632:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537463.928637:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537463.928639:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537463.928640:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537463.928641:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537463.928642:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537463.928643:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537463.928644:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537463.928646:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19263 00000100:00080000:20.0:1584537463.928648:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537463.928657:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537463.928659:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537463.928659:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537463.928661:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537463.928661:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537463.928663:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537463.928663:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537463.928664:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537463.993647:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584537463.993651:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584537463.993652:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584537463.993653:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584537463.993654:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584537463.993656:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584537463.993658:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584537463.993659:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537544 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537463.993693:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078432016 new x1661492078432048 00000800:00000100:0.0:1584537470.141680:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537470.142111:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537470.154183:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537463/real 1584537463] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537550 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537470.154191:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537470.154192:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537470.154194:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537470.154200:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537470.154202:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537470.154202:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537470.154204:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537470.154204:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537470.154207:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537470.154207:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537470.154209:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19269 00000100:00080000:20.0:1584537470.154211:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537470.154220:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537470.154221:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537470.154222:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537470.154223:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537470.154223:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537470.154224:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537470.154225:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537470.154226:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537470.204682:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584537470.204686:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584537470.204687:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584537470.204688:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584537470.204689:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584537470.204691:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584537470.204694:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584537470.204695:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537550 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537470.204724:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078432048 new x1661492078432096 00000800:00020000:2.0:1584537475.916072:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537475.916075:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537475.916077:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537475.916494:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537475.928557:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537470/real 1584537470] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537557 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537475.928566:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537475.928568:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537475.928569:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537475.928586:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537475.928588:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537475.928594:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537475.928597:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537475.928597:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537475.928599:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537475.928599:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537475.928601:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19276 00000100:00080000:20.0:1584537475.928604:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537475.928610:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537475.928611:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537475.928612:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537475.928613:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537475.928614:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537475.928615:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537475.928616:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537475.928617:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537475.970520:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584537475.970525:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584537475.970527:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584537475.970528:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584537475.970530:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584537475.970532:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584537475.970534:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584537475.970535:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537557 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537475.970564:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078432096 new x1661492078432160 00000800:00020000:2.0:1584537481.916089:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537481.916092:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537481.916094:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537481.916499:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537481.928572:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537475/real 1584537475] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537562 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537481.928580:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537481.928581:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537481.928583:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537481.928588:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537481.928591:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537481.928592:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537481.928593:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537481.928594:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537481.928595:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537481.928596:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537481.928598:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19281 00000100:00080000:20.0:1584537481.928600:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537481.928608:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537481.928609:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537481.928610:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537481.928611:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537481.928612:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537481.928613:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537481.928613:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537481.928615:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537481.973650:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584537481.973659:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584537481.973662:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584537481.973664:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584537481.973667:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584537481.973671:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584537481.973675:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584537481.973679:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537562 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537481.973736:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078432160 new x1661492078432208 00000800:00020000:2.0:1584537487.916064:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537487.916067:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537487.916069:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537487.916458:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537487.928572:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537481/real 1584537481] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537568 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537487.928582:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537487.928584:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537487.928587:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537487.928594:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537487.928596:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537487.928597:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537487.928598:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537487.928599:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537487.928601:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537487.928601:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537487.928603:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19287 00000100:00080000:20.0:1584537487.928606:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537487.928621:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537487.928622:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537487.928623:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537487.928624:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537487.928625:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537487.928626:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537487.928627:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537487.928628:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537487.973523:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584537487.973529:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584537487.973531:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584537487.973532:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584537487.973545:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584537487.973547:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584537487.973549:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584537487.973551:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537568 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537487.973584:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078432208 new x1661492078432240 00000800:00000100:0.0:1584537494.141845:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537494.142279:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537494.154380:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537487/real 1584537487] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537574 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537494.154396:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537494.154397:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537494.154399:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537494.154406:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537494.154408:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537494.154408:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537494.154410:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537494.154410:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537494.154412:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537494.154413:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537494.154415:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19293 00000100:00080000:20.0:1584537494.154417:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537494.154433:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537494.154434:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537494.154435:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537494.154436:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537494.154436:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537494.154438:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537494.154438:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537494.154451:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537494.206767:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584537494.206777:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584537494.206781:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584537494.206783:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584537494.206788:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584537494.206792:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584537494.206796:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584537494.206800:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537574 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537494.206848:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078432240 new x1661492078432272 00000800:00020000:6.0:1584537499.916176:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584537499.916179:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584537499.916181:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537499.916748:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537499.928804:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537494/real 1584537494] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537581 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537499.928813:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537499.928814:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537499.928816:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537499.928833:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537499.928835:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537499.928841:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537499.928842:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537499.928843:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537499.928844:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537499.928845:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537499.928847:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19300 00000100:00080000:20.0:1584537499.928849:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537499.928856:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537499.928857:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537499.928858:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537499.928859:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537499.928860:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537499.928861:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537499.928863:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537499.928864:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537500.007713:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584537500.007719:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584537500.007721:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584537500.007722:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584537500.007724:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584537500.007726:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584537500.007727:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584537500.007729:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537581 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537500.007759:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078432272 new x1661492078432352 00000800:00020000:4.0:1584537505.916177:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537505.916179:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537505.916181:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537505.916671:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537505.928723:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537500/real 1584537500] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537587 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537505.928731:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537505.928732:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537505.928733:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537505.928739:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537505.928740:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537505.928741:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537505.928742:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537505.928743:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537505.928744:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537505.928745:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537505.928746:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19305 00000100:00080000:20.0:1584537505.928748:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537505.928757:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537505.928758:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537505.928759:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537505.928760:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537505.928762:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537505.928763:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537505.928763:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537505.928764:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537505.995279:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584537505.995283:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584537505.995284:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584537505.995285:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584537505.995286:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584537505.995288:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584537505.995289:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584537505.995291:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537587 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537505.995322:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078432352 new x1661492078432384 00000800:00020000:4.0:1584537511.916178:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537511.916180:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537511.916182:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537511.916677:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537511.928744:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537505/real 1584537505] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537592 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537511.928752:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537511.928753:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537511.928754:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537511.928760:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537511.928762:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537511.928762:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537511.928763:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537511.928764:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537511.928765:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537511.928766:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537511.928768:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19311 00000100:00080000:20.0:1584537511.928770:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537511.928778:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537511.928780:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537511.928781:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537511.928782:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537511.928783:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537511.928784:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537511.928785:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537511.928786:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537511.983732:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584537511.983736:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584537511.983737:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584537511.983738:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584537511.983740:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584537511.983742:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584537511.983744:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584537511.983746:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537592 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537511.983779:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078432384 new x1661492078432432 00000800:00000100:0.0:1584537518.141021:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537518.141420:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537518.153468:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537511/real 1584537511] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537598 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537518.153476:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537518.153477:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537518.153479:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537518.153484:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537518.153486:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537518.153487:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537518.153488:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537518.153489:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537518.153490:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537518.153491:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537518.153492:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19317 00000100:00080000:20.0:1584537518.153494:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537518.153504:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537518.153505:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537518.153506:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537518.153507:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537518.153508:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537518.153509:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537518.153509:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537518.153511:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537518.199268:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584537518.199272:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584537518.199273:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584537518.199274:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584537518.199276:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584537518.199278:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584537518.199279:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584537518.199281:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537598 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537518.199310:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078432432 new x1661492078432464 00000800:00020000:2.0:1584537523.916122:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537523.916125:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537523.916127:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537523.916624:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537523.928671:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537518/real 1584537518] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537605 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537523.928679:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537523.928680:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537523.928681:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537523.928695:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537523.928696:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537523.928701:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537523.928702:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537523.928703:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537523.928704:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537523.928705:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537523.928707:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19324 00000100:00080000:20.0:1584537523.928709:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537523.928714:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537523.928715:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537523.928716:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537523.928717:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537523.928718:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537523.928719:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537523.928720:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537523.928721:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537523.985326:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584537523.985330:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584537523.985331:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584537523.985332:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584537523.985333:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584537523.985335:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584537523.985337:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584537523.985338:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537605 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537523.985347:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078432464 new x1661492078432528 00000800:00020000:6.0:1584537529.916185:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584537529.916187:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584537529.916189:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537529.916630:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537529.928684:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537523/real 1584537523] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537610 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537529.928692:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537529.928693:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537529.928694:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537529.928700:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537529.928702:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537529.928702:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537529.928704:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537529.928705:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537529.928707:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537529.928707:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537529.928709:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19329 00000100:00080000:20.0:1584537529.928711:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537529.928720:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537529.928721:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537529.928722:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537529.928723:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537529.928724:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537529.928725:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537529.928726:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537529.928727:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537529.970736:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584537529.970740:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584537529.970741:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584537529.970742:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584537529.970744:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584537529.970746:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584537529.970747:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584537529.970749:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537610 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537529.970758:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078432528 new x1661492078432576 00000800:00020000:8.0:1584537535.916053:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584537535.916056:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584537535.916058:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537535.916563:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537535.928614:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537529/real 1584537529] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537616 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537535.928621:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537535.928622:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537535.928624:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537535.928629:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537535.928630:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537535.928631:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537535.928633:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537535.928634:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537535.928635:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537535.928636:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537535.928637:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19335 00000100:00080000:20.0:1584537535.928639:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537535.928648:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537535.928649:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537535.928650:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537535.928651:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537535.928652:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537535.928653:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537535.928653:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537535.928655:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537535.974357:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584537535.974360:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584537535.974362:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584537535.974363:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584537535.974364:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584537535.974366:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584537535.974367:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584537535.974369:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537616 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537535.974377:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078432576 new x1661492078432608 00000800:00000100:0.0:1584537542.141322:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537542.141791:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537542.153873:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537535/real 1584537535] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537622 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537542.153881:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537542.153882:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537542.153883:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537542.153890:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537542.153892:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537542.153893:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537542.153894:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537542.153895:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537542.153896:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537542.153897:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537542.153898:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19341 00000100:00080000:20.0:1584537542.153900:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537542.153910:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537542.153911:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537542.153911:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537542.153912:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537542.153913:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537542.153914:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537542.153915:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537542.153916:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537542.224145:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584537542.224150:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584537542.224152:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584537542.224153:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584537542.224154:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584537542.224156:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584537542.224158:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584537542.224160:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537622 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537542.224193:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078432608 new x1661492078432656 00000800:00020000:4.0:1584537547.916087:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537547.916094:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537547.916098:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537547.916511:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537547.928637:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537542/real 1584537542] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537629 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537547.928647:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537547.928649:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537547.928651:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537547.928669:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537547.928671:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537547.928676:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537547.928678:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537547.928678:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537547.928680:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537547.928681:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537547.928683:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19348 00000100:00080000:20.0:1584537547.928685:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537547.928692:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537547.928693:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537547.928694:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537547.928695:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537547.928696:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537547.928697:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537547.928698:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537547.928699:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537547.977755:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584537547.977761:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584537547.977763:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584537547.977763:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584537547.977766:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584537547.977768:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584537547.977770:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584537547.977771:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537629 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584537547.977805:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078432656 new x1661492078432720 00000800:00020000:2.0:1584537553.916103:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537553.916106:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537553.916108:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537553.916528:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584537553.928582:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537547/real 1584537547] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537634 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584537553.928591:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584537553.928592:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584537553.928594:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584537553.928602:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537553.928604:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537553.928605:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537553.928606:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537553.928607:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537553.928608:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584537553.928609:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584537553.928611:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19353 00000100:00080000:22.0:1584537553.928613:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584537553.928628:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537553.928629:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537553.928630:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537553.928631:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537553.928633:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537553.928635:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537553.928635:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537553.928636:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537553.971968:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584537553.971974:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584537553.971977:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584537553.971978:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584537553.971979:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584537553.971982:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584537553.971983:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584537553.971985:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537634 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584537553.971994:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078432720 new x1661492078432752 00000800:00020000:6.0:1584537559.916186:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584537559.916189:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584537559.916191:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537559.916569:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584537559.928622:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537553/real 1584537553] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537640 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584537559.928631:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584537559.928632:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584537559.928633:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584537559.928639:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537559.928640:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537559.928641:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537559.928642:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537559.928643:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537559.928645:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584537559.928645:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584537559.928647:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19359 00000100:00080000:22.0:1584537559.928649:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584537559.928659:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537559.928661:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537559.928662:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537559.928663:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537559.928664:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537559.928665:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537559.928666:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537559.928667:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537559.966439:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584537559.966444:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584537559.966446:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584537559.966447:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584537559.966448:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584537559.966450:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584537559.966452:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584537559.966454:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537640 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584537559.966483:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078432752 new x1661492078432800 00000800:00000100:0.0:1584537566.141287:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537566.141720:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584537566.153769:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537559/real 1584537559] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537646 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584537566.153777:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584537566.153778:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584537566.153779:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584537566.153785:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537566.153787:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537566.153788:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537566.153789:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537566.153789:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537566.153791:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584537566.153791:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584537566.153793:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19365 00000100:00080000:22.0:1584537566.153795:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584537566.153804:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537566.153806:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537566.153806:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537566.153807:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537566.153808:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537566.153809:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537566.153810:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537566.153811:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584537566.207594:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584537566.207602:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584537566.207605:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584537566.207607:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584537566.207612:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584537566.207616:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584537566.207619:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584537566.207623:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537646 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584537566.207677:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078432800 new x1661492078432832 00000800:00020000:4.0:1584537571.916043:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537571.916045:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537571.916047:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537571.916456:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584537571.928521:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537566/real 1584537566] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537653 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584537571.928529:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584537571.928530:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584537571.928531:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584537571.928546:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537571.928547:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537571.928553:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537571.928554:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537571.928555:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537571.928556:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584537571.928557:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584537571.928559:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19372 00000100:00080000:22.0:1584537571.928561:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584537571.928566:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537571.928568:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537571.928568:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537571.928570:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537571.928570:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537571.928571:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537571.928572:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584537571.928573:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537571.976546:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584537571.976550:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584537571.976552:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584537571.976553:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584537571.976555:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584537571.976557:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584537571.976559:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584537571.976561:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537653 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:0.0:1584537571.976570:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078432832 new x1661492078432912 00000800:00020000:6.0:1584537577.916169:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584537577.916172:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584537577.916174:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537577.916580:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584537577.928635:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537571/real 1584537571] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537658 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584537577.928643:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584537577.928644:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584537577.928646:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584537577.928650:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584537577.928652:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537577.928652:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584537577.928654:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537577.928654:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584537577.928656:0:4532:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584537577.928656:0:4532:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584537577.928658:0:4532:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19377 00000100:00080000:2.0:1584537577.928661:0:4532:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584537577.928670:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584537577.928671:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537577.928672:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584537577.928673:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537577.928673:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584537577.928674:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537577.928675:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584537577.928676:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537577.993107:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584537577.993111:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584537577.993113:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584537577.993114:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584537577.993115:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584537577.993117:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584537577.993118:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584537577.993120:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537658 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584537577.993129:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078432912 new x1661492078432944 00000800:00020000:2.0:1584537583.916166:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537583.916169:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537583.916171:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537583.916580:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584537583.928635:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537577/real 1584537577] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537664 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584537583.928644:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584537583.928645:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584537583.928646:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584537583.928651:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537583.928653:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537583.928654:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537583.928655:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537583.928655:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537583.928657:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584537583.928657:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584537583.928659:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19383 00000100:00080000:4.0:1584537583.928661:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584537583.928669:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537583.928671:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537583.928671:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537583.928672:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537583.928673:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537583.928674:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537583.928675:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537583.928676:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537586.498008:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584537586.498012:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584537586.498013:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584537586.498014:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584537586.498015:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584537586.498017:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584537586.498019:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584537586.498021:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537664 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584537586.498066:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078432944 new x1661492078432976 00000800:00020000:2.0:1584537591.916087:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537591.916089:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537591.916091:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537591.916478:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584537591.928535:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537586/real 1584537586] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537675 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584537591.928543:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584537591.928544:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584537591.928545:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584537591.928551:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537591.928553:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537591.928553:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537591.928554:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537591.928555:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537591.928556:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584537591.928557:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584537591.928559:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19389 00000100:00080000:4.0:1584537591.928561:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584537591.928569:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537591.928571:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537591.928571:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537591.928573:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537591.928573:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537591.928574:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537591.928576:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537591.928577:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537591.976068:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584537591.976072:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584537591.976074:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584537591.976075:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584537591.976077:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584537591.976079:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584537591.976081:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584537591.976082:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537675 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584537591.976091:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078432976 new x1661492078433024 00000800:00020000:8.0:1584537597.916063:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584537597.916066:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584537597.916068:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537597.916690:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584537597.928743:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537591/real 1584537591] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537678 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584537597.928751:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584537597.928752:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584537597.928754:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584537597.928768:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537597.928769:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537597.928774:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537597.928776:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537597.928776:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537597.928778:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584537597.928778:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584537597.928780:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19397 00000100:00080000:6.0:1584537597.928782:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584537597.928787:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537597.928788:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537597.928789:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537597.928790:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537597.928790:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537597.928791:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537597.928792:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537597.928793:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537597.966639:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584537597.966643:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584537597.966644:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584537597.966645:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584537597.966646:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584537597.966648:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584537597.966650:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584537597.966652:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537678 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584537597.966660:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078433024 new x1661492078433088 00000800:00020000:2.0:1584537603.916112:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537603.916116:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537603.916118:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537603.916534:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584537603.928615:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537597/real 1584537597] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537684 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584537603.928626:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584537603.928627:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584537603.928629:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584537603.928637:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537603.928639:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537603.928640:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537603.928641:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537603.928642:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537603.928643:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584537603.928644:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584537603.928646:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19403 00000100:00080000:6.0:1584537603.928650:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584537603.928664:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537603.928666:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537603.928666:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537603.928667:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537603.928668:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537603.928669:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537603.928670:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537603.928671:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537603.971963:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584537603.971970:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584537603.971972:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584537603.971973:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584537603.971976:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584537603.971978:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584537603.971980:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584537603.971981:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537684 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584537603.972012:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078433088 new x1661492078433136 00000800:00000100:0.0:1584537610.141304:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537610.141746:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584537610.153793:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537603/real 1584537603] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537690 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584537610.153802:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584537610.153803:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584537610.153805:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584537610.153811:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537610.153813:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537610.153814:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537610.153815:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537610.153816:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537610.153817:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584537610.153818:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584537610.153820:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19409 00000100:00080000:6.0:1584537610.153823:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584537610.153837:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537610.153838:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537610.153839:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537610.153840:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537610.153841:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537610.153842:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537610.153844:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537610.153845:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537610.162888:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584537610.162894:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584537610.162896:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584537610.162897:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584537610.162899:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584537610.162902:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584537610.162903:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584537610.162905:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537690 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584537610.162935:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078433136 new x1661492078433168 00000800:00020000:4.0:1584537615.916081:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537615.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537615.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537615.916554:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584537615.928597:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537610/real 1584537610] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537697 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584537615.928605:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584537615.928606:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584537615.928607:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584537615.928612:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537615.928614:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537615.928614:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537615.928616:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537615.928616:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537615.928618:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584537615.928618:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584537615.928620:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19416 00000100:00080000:6.0:1584537615.928622:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584537615.928631:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537615.928633:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537615.928633:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537615.928634:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537615.928636:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537615.928637:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537615.928638:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537615.928639:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537615.936370:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584537615.936374:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584537615.936376:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584537615.936377:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584537615.936378:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584537615.936380:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584537615.936381:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584537615.936383:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537697 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584537615.936412:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078433168 new x1661492078433200 00000800:00020000:4.0:1584537621.916100:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537621.916103:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537621.916105:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537621.916640:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584537621.928719:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537615/real 1584537615] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537702 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584537621.928727:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584537621.928728:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584537621.928729:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584537621.928744:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537621.928745:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537621.928751:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537621.928752:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537621.928753:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537621.928754:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584537621.928755:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584537621.928756:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19421 00000100:00080000:6.0:1584537621.928759:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584537621.928764:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537621.928765:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537621.928767:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537621.928768:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537621.928769:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537621.928770:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537621.928771:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537621.928772:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537621.936632:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584537621.936636:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584537621.936637:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584537621.936638:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584537621.936640:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584537621.936641:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584537621.936643:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584537621.936645:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537702 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584537621.936653:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078433200 new x1661492078433280 00000800:00020000:6.0:1584537627.916055:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584537627.916057:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584537627.916059:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537627.916438:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584537627.928513:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537621/real 1584537621] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537708 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584537627.928520:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584537627.928522:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584537627.928523:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584537627.928529:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537627.928530:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537627.928531:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537627.928532:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537627.928533:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537627.928534:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584537627.928535:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584537627.928537:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19427 00000100:00080000:6.0:1584537627.928539:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584537627.928548:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537627.928549:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537627.928550:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537627.928551:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537627.928552:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537627.928553:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537627.928553:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537627.928554:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537627.936401:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584537627.936405:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584537627.936407:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584537627.936407:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584537627.936410:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584537627.936411:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584537627.936413:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584537627.936415:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537708 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584537627.936424:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078433280 new x1661492078433312 00000800:00000100:0.0:1584537634.141104:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537634.141514:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584537634.153570:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537627/real 1584537627] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537714 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584537634.153577:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584537634.153578:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584537634.153580:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584537634.153585:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537634.153587:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537634.153588:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537634.153589:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537634.153590:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537634.153592:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584537634.153593:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584537634.153594:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19433 00000100:00080000:6.0:1584537634.153596:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584537634.153605:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537634.153606:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537634.153607:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537634.153608:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537634.153608:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537634.153609:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537634.153610:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537634.153611:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537634.161266:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584537634.161270:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584537634.161271:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584537634.161272:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584537634.161273:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584537634.161275:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584537634.161277:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584537634.161279:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537714 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584537634.161287:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078433312 new x1661492078433360 00000800:00020000:0.0:1584537639.916068:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584537639.916071:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584537639.916073:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537639.916453:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584537639.928502:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537634/real 1584537634] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537721 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584537639.928510:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584537639.928511:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584537639.928513:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584537639.928518:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537639.928520:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537639.928521:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537639.928523:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537639.928524:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537639.928525:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584537639.928526:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584537639.928527:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19440 00000100:00080000:6.0:1584537639.928529:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584537639.928538:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537639.928539:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537639.928540:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537639.928541:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537639.928541:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537639.928543:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537639.928543:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537639.928544:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537639.936389:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584537639.936393:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584537639.936394:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584537639.936395:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584537639.936396:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584537639.936398:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584537639.936400:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584537639.936402:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537721 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584537639.936431:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078433360 new x1661492078433392 00000800:00020000:2.0:1584537645.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537645.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537645.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537645.916484:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584537645.928562:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537639/real 1584537639] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537726 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584537645.928570:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584537645.928571:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584537645.928572:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584537645.928586:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537645.928591:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537645.928595:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537645.928597:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537645.928597:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537645.928599:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584537645.928599:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584537645.928601:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19445 00000100:00080000:6.0:1584537645.928603:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584537645.928608:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537645.928609:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537645.928610:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537645.928611:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537645.928611:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537645.928613:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537645.928613:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537645.928615:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537645.936582:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584537645.936586:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584537645.936588:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584537645.936588:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584537645.936591:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584537645.936592:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584537645.936594:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584537645.936595:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537726 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584537645.936624:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078433392 new x1661492078433456 00000800:00020000:10.0:1584537651.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:10.0:1584537651.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:10.0:1584537651.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537651.916550:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584537651.928603:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537645/real 1584537645] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537732 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584537651.928611:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584537651.928612:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584537651.928615:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584537651.928620:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537651.928622:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537651.928623:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537651.928624:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537651.928625:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537651.928626:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584537651.928627:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584537651.928628:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19451 00000100:00080000:6.0:1584537651.928630:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584537651.928639:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537651.928641:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537651.928641:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537651.928643:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537651.928643:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537651.928644:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537651.928645:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537651.928646:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537651.936579:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584537651.936583:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584537651.936584:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584537651.936585:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584537651.936586:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584537651.936588:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584537651.936590:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584537651.936591:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537732 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584537651.936620:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078433456 new x1661492078433504 00000800:00000100:0.0:1584537658.141375:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537658.141775:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584537658.153824:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537651/real 1584537651] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537738 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584537658.153836:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584537658.153837:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584537658.153838:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584537658.153843:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537658.153844:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537658.153845:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537658.153846:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537658.153847:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537658.153848:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584537658.153849:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584537658.153850:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19457 00000100:00080000:6.0:1584537658.153853:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584537658.153861:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537658.153863:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537658.153863:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537658.153864:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537658.153865:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537658.153866:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537658.153867:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537658.153868:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537658.161506:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584537658.161510:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584537658.161511:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584537658.161512:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584537658.161513:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584537658.161515:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584537658.161517:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584537658.161518:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537738 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584537658.161526:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078433504 new x1661492078433536 00000800:00020000:2.0:1584537663.916085:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537663.916089:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537663.916091:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537663.916506:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584537663.928587:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537658/real 1584537658] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537745 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584537663.928596:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584537663.928598:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584537663.928599:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584537663.928607:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537663.928609:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537663.928610:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537663.928611:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537663.928612:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537663.928614:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584537663.928615:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584537663.928617:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19464 00000100:00080000:6.0:1584537663.928619:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584537663.928629:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537663.928631:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537663.928631:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537663.928632:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537663.928633:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537663.928634:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584537663.928636:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584537663.928637:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537663.936446:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584537663.936451:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584537663.936453:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584537663.936454:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584537663.936457:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584537663.936459:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584537663.936460:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584537663.936462:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537745 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584537663.936491:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078433536 new x1661492078433584 00000800:00020000:4.0:1584537669.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537669.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537669.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537669.916655:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537669.928735:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537663/real 1584537663] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537750 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537669.928746:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537669.928747:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537669.928749:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537669.928768:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537669.928770:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537669.928775:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537669.928777:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537669.928777:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537669.928779:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537669.928780:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537669.928782:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19469 00000100:00080000:8.0:1584537669.928785:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537669.928792:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537669.928794:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537669.928794:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537669.928795:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537669.928797:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537669.928798:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537669.928799:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537669.928800:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537669.937023:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584537669.937052:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584537669.937054:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584537669.937055:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584537669.937057:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584537669.937060:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584537669.937061:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584537669.937063:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537750 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584537669.937073:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078433584 new x1661492078433648 00000800:00020000:4.0:1584537675.916101:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537675.916104:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537675.916106:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537675.916578:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537675.928657:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537669/real 1584537669] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537756 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537675.928667:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537675.928668:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537675.928670:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537675.928678:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537675.928680:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537675.928681:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537675.928682:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537675.928683:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537675.928684:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537675.928685:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537675.928687:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19475 00000100:00080000:8.0:1584537675.928690:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537675.928704:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537675.928706:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537675.928708:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537675.928709:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537675.928710:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537675.928711:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537675.928712:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537675.928713:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537675.936435:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584537675.936442:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584537675.936444:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584537675.936445:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584537675.936447:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584537675.936449:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584537675.936451:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584537675.936453:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537756 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584537675.936482:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078433648 new x1661492078433680 00000800:00000100:0.0:1584537682.140828:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537682.141330:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537682.153400:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537675/real 1584537675] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537762 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537682.153408:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537682.153409:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537682.153411:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537682.153417:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537682.153419:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537682.153420:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537682.153421:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537682.153422:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537682.153423:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537682.153424:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537682.153426:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19481 00000100:00080000:8.0:1584537682.153428:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537682.153439:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537682.153440:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537682.153441:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537682.153442:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537682.153443:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537682.153444:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537682.153445:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537682.153446:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537682.161321:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584537682.161327:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584537682.161329:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584537682.161330:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584537682.161331:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584537682.161334:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584537682.161335:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584537682.161337:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537762 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584537682.161367:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078433680 new x1661492078433728 00000800:00020000:0.0:1584537687.916066:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584537687.916069:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584537687.916071:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537687.916481:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537687.928560:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537682/real 1584537682] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537769 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537687.928569:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537687.928570:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537687.928571:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537687.928578:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537687.928580:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537687.928581:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537687.928582:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537687.928582:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537687.928584:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537687.928585:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537687.928587:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19488 00000100:00080000:8.0:1584537687.928590:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537687.928600:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537687.928601:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537687.928602:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537687.928603:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537687.928604:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537687.928605:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537687.928606:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537687.928607:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537687.936518:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584537687.936523:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584537687.936525:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584537687.936526:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584537687.936529:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584537687.936531:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584537687.936533:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584537687.936535:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537769 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584537687.936564:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078433728 new x1661492078433760 00000800:00020000:2.0:1584537693.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537693.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537693.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537693.916496:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537693.928574:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537687/real 1584537687] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537774 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537693.928582:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537693.928583:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537693.928585:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537693.928599:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537693.928601:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537693.928606:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537693.928607:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537693.928608:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537693.928610:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537693.928610:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537693.928612:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19493 00000100:00080000:8.0:1584537693.928614:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537693.928620:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537693.928621:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537693.928622:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537693.928623:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537693.928624:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537693.928625:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537693.928626:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537693.928627:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537693.936518:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584537693.936523:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584537693.936525:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584537693.936526:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584537693.936527:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584537693.936529:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584537693.936531:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584537693.936532:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537774 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584537693.936540:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078433760 new x1661492078433840 00000800:00020000:2.0:1584537699.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537699.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537699.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537699.916575:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584537699.928627:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537693/real 1584537693] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537780 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584537699.928634:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584537699.928635:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584537699.928637:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584537699.928643:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537699.928644:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537699.928645:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537699.928646:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537699.928647:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537699.928648:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584537699.928649:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584537699.928650:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19499 00000100:00080000:10.0:1584537699.928652:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584537699.928661:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537699.928662:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537699.928664:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537699.928665:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537699.928666:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537699.928667:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537699.928668:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537699.928669:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537699.936741:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584537699.936745:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584537699.936747:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584537699.936747:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584537699.936749:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584537699.936751:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584537699.936752:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584537699.936754:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537780 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584537699.936788:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078433840 new x1661492078433872 00000800:00000100:0.0:1584537706.141204:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537706.141697:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584537706.153755:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537699/real 1584537699] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537786 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584537706.153763:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584537706.153764:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584537706.153765:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584537706.153771:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537706.153772:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537706.153773:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537706.153774:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537706.153775:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537706.153776:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584537706.153777:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584537706.153778:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19505 00000100:00080000:10.0:1584537706.153780:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584537706.153801:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537706.153802:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537706.153802:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537706.153804:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537706.153804:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537706.153805:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537706.153806:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537706.153807:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537706.162066:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584537706.162070:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584537706.162071:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584537706.162072:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584537706.162074:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584537706.162076:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584537706.162078:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584537706.162079:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537786 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584537706.162110:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078433872 new x1661492078433904 00000800:00020000:2.0:1584537711.916072:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537711.916074:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537711.916076:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537711.916492:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584537711.928544:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537706/real 1584537706] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537793 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584537711.928552:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584537711.928553:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584537711.928554:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584537711.928560:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537711.928561:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537711.928562:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537711.928563:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537711.928564:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537711.928565:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584537711.928566:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584537711.928569:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19512 00000100:00080000:10.0:1584537711.928571:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584537711.928579:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537711.928581:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537711.928581:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537711.928583:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537711.928583:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537711.928584:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537711.928585:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537711.928586:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537711.936342:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584537711.936347:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584537711.936348:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584537711.936349:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584537711.936350:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584537711.936352:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584537711.936354:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584537711.936355:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537793 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584537711.936384:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078433904 new x1661492078433952 00000800:00020000:2.0:1584537717.916069:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537717.916072:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537717.916074:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537717.916518:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584537717.928571:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537711/real 1584537712] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537798 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584537717.928579:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584537717.928580:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584537717.928581:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584537717.928595:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537717.928597:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537717.928603:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537717.928604:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537717.928605:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537717.928607:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584537717.928607:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584537717.928609:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19517 00000100:00080000:10.0:1584537717.928611:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584537717.928616:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537717.928618:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537717.928618:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537717.928619:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537717.928620:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537717.928621:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537717.928622:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537717.928623:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537717.936441:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537717.936446:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537717.936447:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537717.936448:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537717.936449:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537717.936451:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537717.936453:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537717.936454:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537798 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584537717.936483:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078433952 new x1661492078434016 00000800:00020000:6.0:1584537723.916122:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584537723.916126:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584537723.916128:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537723.916570:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584537723.928628:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537717/real 1584537717] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537804 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584537723.928638:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584537723.928639:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584537723.928641:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584537723.928648:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537723.928650:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537723.928651:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537723.928652:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537723.928653:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537723.928654:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584537723.928655:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584537723.928657:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19523 00000100:00080000:14.0:1584537723.928660:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584537723.928671:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537723.928673:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537723.928673:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537723.928676:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537723.928676:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537723.928678:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537723.928678:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537723.928680:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537723.936839:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537723.936845:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537723.936847:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537723.936848:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537723.936850:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537723.936852:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537723.936853:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537723.936855:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537804 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584537723.936884:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078434016 new x1661492078434064 00000800:00000100:0.0:1584537730.140837:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537730.141293:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584537730.153348:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537723/real 1584537723] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537810 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584537730.153356:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584537730.153357:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584537730.153359:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584537730.153365:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537730.153367:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537730.153368:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537730.153369:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537730.153370:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537730.153371:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584537730.153372:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584537730.153374:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19529 00000100:00080000:14.0:1584537730.153376:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584537730.153387:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537730.153389:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537730.153390:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537730.153391:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537730.153392:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537730.153393:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537730.153394:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537730.153395:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537730.161460:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537730.161466:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537730.161468:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537730.161470:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537730.161472:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537730.161474:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537730.161475:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537730.161477:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537810 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584537730.161509:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078434064 new x1661492078434096 00000800:00020000:2.0:1584537735.916069:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537735.916072:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537735.916074:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537735.916488:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584537735.928540:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537730/real 1584537730] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537817 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584537735.928548:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584537735.928549:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584537735.928550:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584537735.928556:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537735.928558:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537735.928559:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537735.928560:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537735.928561:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537735.928563:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584537735.928563:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584537735.928565:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19536 00000100:00080000:14.0:1584537735.928568:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584537735.928578:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537735.928580:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537735.928580:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537735.928582:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537735.928582:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537735.928584:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537735.928584:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537735.928585:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584537735.936621:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584537735.936626:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584537735.936628:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584537735.936629:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584537735.936630:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584537735.936632:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584537735.936634:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584537735.936636:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537817 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584537735.936665:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078434096 new x1661492078434128 00000800:00020000:2.0:1584537741.916069:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537741.916072:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537741.916074:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537741.916562:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584537741.928615:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537735/real 1584537735] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537822 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584537741.928624:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584537741.928625:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584537741.928626:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584537741.928642:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537741.928644:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537741.928650:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537741.928652:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537741.928652:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537741.928654:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584537741.928656:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584537741.928657:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19541 00000100:00080000:14.0:1584537741.928660:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584537741.928666:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537741.928667:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537741.928668:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537741.928669:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537741.928670:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537741.928671:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537741.928672:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537741.928673:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537741.936470:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584537741.936475:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584537741.936477:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584537741.936478:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584537741.936481:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584537741.936483:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584537741.936485:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584537741.936487:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537822 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584537741.936516:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078434128 new x1661492078434208 00000800:00020000:2.0:1584537747.916059:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537747.916062:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537747.916063:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537747.916556:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584537747.928611:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537741/real 1584537741] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537828 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584537747.928618:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584537747.928619:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584537747.928620:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584537747.928626:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537747.928628:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537747.928629:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537747.928630:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537747.928632:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537747.928633:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584537747.928634:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584537747.928635:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19547 00000100:00080000:14.0:1584537747.928637:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584537747.928646:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537747.928647:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537747.928648:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537747.928649:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537747.928650:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537747.928651:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537747.928652:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537747.928653:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537747.936721:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584537747.936725:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584537747.936727:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584537747.936727:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584537747.936729:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584537747.936731:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584537747.936732:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584537747.936734:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537828 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584537747.936763:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078434208 new x1661492078434240 00000800:00000100:0.0:1584537754.141181:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537754.141596:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584537754.153645:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537747/real 1584537747] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537834 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584537754.197429:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584537754.197430:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584537754.197432:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584537754.197447:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537754.197448:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537754.197450:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537754.197451:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537754.197452:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537754.197453:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584537754.197453:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584537754.197455:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19553 00000100:00080000:14.0:1584537754.197457:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584537754.197466:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537754.197467:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537754.197468:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537754.197469:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537754.197469:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537754.197471:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537754.197471:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537754.197472:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537754.205452:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584537754.205457:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584537754.205458:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584537754.205459:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584537754.205460:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584537754.205462:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584537754.205464:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584537754.205466:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537834 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584537754.205495:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078434240 new x1661492078434288 00000800:00020000:2.0:1584537759.916083:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537759.916086:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537759.916088:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537759.916497:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584537759.928549:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537754/real 1584537754] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537841 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584537759.928557:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584537759.928558:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584537759.928559:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584537759.928565:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537759.928567:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537759.928568:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537759.928569:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537759.928569:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537759.928571:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584537759.928571:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584537759.928573:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19560 00000100:00080000:14.0:1584537759.928575:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584537759.928583:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537759.928584:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537759.928585:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537759.928586:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537759.928587:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537759.928588:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537759.928588:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537759.928590:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537759.936399:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584537759.936403:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584537759.936405:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584537759.936405:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584537759.936408:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584537759.936410:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584537759.936411:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584537759.936413:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537841 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584537759.936421:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078434288 new x1661492078434320 00000800:00020000:2.0:1584537765.916050:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537765.916053:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537765.916055:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537765.916466:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584537765.928517:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537759/real 1584537759] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537846 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584537765.928524:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584537765.928526:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584537765.928527:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584537765.928540:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537765.928542:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537765.928547:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537765.928548:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537765.928549:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537765.928550:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584537765.928551:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584537765.928552:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19565 00000100:00080000:14.0:1584537765.928554:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584537765.928559:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537765.928560:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537765.928561:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537765.928562:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537765.928563:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537765.928564:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537765.928564:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537765.928566:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537765.937674:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584537765.937678:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584537765.937680:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584537765.937681:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584537765.937682:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584537765.937684:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584537765.937685:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584537765.937687:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537846 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584537765.937695:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078434320 new x1661492078434384 00000800:00020000:4.0:1584537771.916059:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537771.916062:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537771.916064:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537771.916477:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584537771.928530:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537765/real 1584537765] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537852 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584537771.928538:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584537771.928540:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584537771.928541:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584537771.928547:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537771.928548:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537771.928549:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537771.928550:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537771.928551:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537771.928552:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584537771.928553:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584537771.928554:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19571 00000100:00080000:14.0:1584537771.928557:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584537771.928566:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537771.928567:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537771.928568:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537771.928569:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537771.928569:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537771.928570:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537771.928571:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537771.928572:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537771.958816:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584537771.958820:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584537771.958822:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584537771.958823:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584537771.958824:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584537771.958826:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584537771.958827:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584537771.958829:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537852 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584537771.958837:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078434384 new x1661492078434432 00000800:00000100:0.0:1584537778.141467:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537778.141877:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584537778.153926:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537771/real 1584537771] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537858 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584537778.153933:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584537778.153934:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584537778.153935:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584537778.153941:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537778.153942:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537778.153943:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537778.153944:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537778.153945:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537778.153946:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584537778.153947:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584537778.153949:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19577 00000100:00080000:14.0:1584537778.153951:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584537778.153960:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537778.153961:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537778.153962:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537778.153963:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537778.153964:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537778.153965:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537778.153966:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537778.153968:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537778.209328:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584537778.209333:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584537778.209334:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584537778.209335:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584537778.209337:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584537778.209339:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584537778.209340:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584537778.209342:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537858 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584537778.209372:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078434432 new x1661492078434464 00000800:00020000:2.0:1584537783.916119:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537783.916123:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537783.916125:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537783.916516:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584537783.928573:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537778/real 1584537778] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537865 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584537783.928583:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584537783.928584:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584537783.928586:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584537783.928592:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537783.928594:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537783.928595:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537783.928596:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537783.928597:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537783.928599:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584537783.928600:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584537783.928601:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19584 00000100:00080000:14.0:1584537783.928604:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584537783.928617:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537783.928618:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537783.928619:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537783.928620:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537783.928622:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537783.928624:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537783.928624:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537783.928626:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537783.952432:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584537783.952439:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584537783.952441:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584537783.952442:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584537783.952444:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584537783.952446:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584537783.952448:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584537783.952450:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537865 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584537783.952458:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078434464 new x1661492078434512 00000800:00020000:2.0:1584537789.916040:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537789.916043:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537789.916045:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537789.916535:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584537789.928584:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537783/real 1584537783] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537870 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584537789.928593:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584537789.928594:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584537789.928596:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584537789.928616:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537789.928618:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537789.928623:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537789.928625:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537789.928625:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537789.928627:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584537789.928628:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584537789.928630:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19589 00000100:00080000:14.0:1584537789.928632:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584537789.928639:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537789.928641:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537789.928642:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537789.928643:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537789.928644:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537789.928645:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537789.928646:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537789.928647:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537789.982275:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584537789.982282:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584537789.982284:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584537789.982285:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584537789.982287:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584537789.982289:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584537789.982291:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584537789.982293:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537870 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584537789.982301:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078434512 new x1661492078434576 00000800:00020000:0.0:1584537795.916051:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584537795.916054:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584537795.916056:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537795.916439:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584537795.928487:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537789/real 1584537789] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537876 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584537795.928494:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584537795.928495:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584537795.928497:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584537795.928502:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537795.928504:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537795.928505:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537795.928506:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537795.928506:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537795.928508:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584537795.928509:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584537795.928510:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19595 00000100:00080000:14.0:1584537795.928512:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584537795.928522:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537795.928523:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537795.928524:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537795.928525:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537795.928526:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537795.928527:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537795.928528:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537795.928529:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537795.967875:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584537795.967879:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584537795.967880:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584537795.967881:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584537795.967883:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584537795.967885:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584537795.967887:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584537795.967889:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537876 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584537795.967897:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078434576 new x1661492078434608 00000800:00000100:0.0:1584537802.141024:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537802.141557:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584537802.153605:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537795/real 1584537795] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537882 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584537802.153613:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584537802.153614:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584537802.153615:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584537802.153622:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537802.153624:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537802.153624:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537802.153625:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537802.153626:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537802.153628:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584537802.153628:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584537802.153631:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19601 00000100:00080000:14.0:1584537802.153633:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584537802.153642:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537802.153643:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537802.153644:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537802.153645:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537802.153646:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537802.153647:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537802.153648:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537802.153649:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537802.199705:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584537802.199709:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584537802.199711:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584537802.199712:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584537802.199713:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584537802.199715:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584537802.199717:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584537802.199719:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537882 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584537802.199727:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078434608 new x1661492078434656 00000800:00020000:4.0:1584537807.916188:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537807.916191:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537807.916193:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537807.916575:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584537807.928627:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537802/real 1584537802] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537889 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584537807.928635:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584537807.928636:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584537807.928637:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584537807.928643:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537807.928645:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537807.928646:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537807.928647:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537807.928648:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537807.928650:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584537807.928650:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584537807.928652:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19608 00000100:00080000:14.0:1584537807.928654:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584537807.928663:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537807.928664:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537807.928664:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537807.928666:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537807.928666:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537807.928668:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537807.928668:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537807.928669:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537807.982239:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584537807.982243:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584537807.982245:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584537807.982245:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584537807.982247:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584537807.982249:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584537807.982250:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584537807.982252:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537889 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584537807.982260:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078434656 new x1661492078434688 00000800:00020000:4.0:1584537813.916051:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537813.916054:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537813.916056:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537813.916474:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584537813.928535:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537807/real 1584537807] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537894 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584537813.928544:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584537813.928545:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584537813.928546:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584537813.928561:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537813.928563:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537813.928570:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537813.928572:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537813.928572:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537813.928574:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584537813.928575:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584537813.928576:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19613 00000100:00080000:14.0:1584537813.928579:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584537813.928586:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537813.928587:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537813.928588:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537813.928589:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537813.928590:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537813.928591:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537813.928591:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537813.928593:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537813.977537:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584537813.977543:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584537813.977545:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584537813.977546:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584537813.977548:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584537813.977551:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584537813.977552:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584537813.977554:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537894 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584537813.977563:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078434688 new x1661492078434768 00000800:00020000:2.0:1584537819.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537819.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537819.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537819.916555:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584537819.928609:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537813/real 1584537813] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537900 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584537819.928616:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584537819.928617:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584537819.928619:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584537819.928624:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537819.928627:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537819.928627:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537819.928629:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537819.928629:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537819.928631:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584537819.928631:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584537819.928633:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19619 00000100:00080000:14.0:1584537819.928635:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584537819.928643:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537819.928645:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537819.928645:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537819.928646:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537819.928647:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537819.928648:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537819.928649:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537819.928650:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537819.984891:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584537819.984895:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584537819.984897:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584537819.984898:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584537819.984899:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584537819.984901:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584537819.984902:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584537819.984904:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537900 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584537819.984913:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078434768 new x1661492078434800 00000800:00000100:0.0:1584537826.141317:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537826.141811:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584537826.153865:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537819/real 1584537819] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537906 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584537826.153873:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584537826.179264:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584537826.179266:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584537826.179269:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537826.179271:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537826.179272:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537826.179273:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537826.179274:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537826.179275:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584537826.179275:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584537826.179277:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19625 00000100:00080000:14.0:1584537826.179279:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584537826.179287:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537826.179288:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537826.179289:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537826.179290:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537826.179291:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537826.179292:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584537826.179292:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584537826.179294:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537826.214819:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584537826.214823:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584537826.214824:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584537826.214825:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584537826.214826:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584537826.214828:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584537826.214830:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584537826.232656:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537906 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584537826.232694:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078434800 new x1661492078434832 00000800:00020000:2.0:1584537831.916103:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537831.916106:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537831.916107:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537831.916575:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584537831.928633:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537826/real 1584537826] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537913 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584537831.928641:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584537831.928642:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584537831.928643:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584537831.928650:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537831.928652:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537831.928653:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537831.928654:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537831.928656:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537831.928657:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584537831.928658:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584537831.928659:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19632 00000100:00080000:16.0:1584537831.928661:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584537831.928671:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537831.928673:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537831.928673:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537831.928674:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537831.928675:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537831.928676:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537831.928677:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537831.928678:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537831.976341:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584537831.976345:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584537831.976347:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584537831.976348:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584537831.976350:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584537831.976352:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584537831.976353:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584537831.976355:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537913 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584537831.976387:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078434832 new x1661492078434880 00000800:00020000:4.0:1584537837.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537837.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537837.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537837.916519:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584537837.928571:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537831/real 1584537831] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537918 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584537837.928579:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584537837.928580:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584537837.928581:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584537837.928596:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537837.928597:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537837.928604:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537837.928605:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537837.928606:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537837.928607:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584537837.928608:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584537837.928609:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19637 00000100:00080000:16.0:1584537837.928611:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584537837.928617:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537837.928618:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537837.928619:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537837.928620:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537837.928621:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537837.928622:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584537837.928622:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584537837.928623:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537837.967280:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584537837.967285:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584537837.967286:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584537837.967287:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584537837.967288:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584537837.967290:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584537837.967292:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584537837.967294:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537918 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537837.967302:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078434880 new x1661492078434944 00000800:00020000:2.0:1584537843.916177:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537843.916180:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537843.916182:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537843.916632:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537843.928689:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537837/real 1584537837] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537924 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537843.928698:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537843.928699:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537843.928701:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537843.928707:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537843.928709:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537843.928710:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537843.928711:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537843.928712:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537843.928713:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537843.928714:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537843.928716:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19643 00000100:00080000:20.0:1584537843.928718:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537843.928727:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537843.928729:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537843.928729:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537843.928732:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537843.928732:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537843.928733:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537843.928734:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537843.928735:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537843.978820:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584537843.978825:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584537843.978827:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584537843.978828:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584537843.978830:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584537843.978832:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584537843.978834:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584537843.978836:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537924 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537843.978844:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078434944 new x1661492078434992 00000800:00000100:0.0:1584537850.141711:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537850.142126:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537850.154180:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537843/real 1584537843] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537930 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537850.154189:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537850.154190:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537850.154191:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537850.154197:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537850.154199:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537850.154200:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537850.154201:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537850.154202:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537850.154204:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537850.154204:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537850.154206:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19649 00000100:00080000:20.0:1584537850.154208:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537850.154218:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537850.154221:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537850.154221:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537850.154222:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537850.154223:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537850.154224:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537850.154225:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537850.154226:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537850.193968:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584537850.193973:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584537850.193975:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584537850.193976:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584537850.193978:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584537850.193980:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584537850.193981:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584537850.193983:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537930 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537850.194011:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078434992 new x1661492078435024 00000800:00020000:2.0:1584537855.916067:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537855.916070:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537855.916072:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537855.916575:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537855.928668:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537850/real 1584537850] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537937 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537855.928676:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537855.928677:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537855.928679:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537855.928685:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537855.928687:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537855.928688:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537855.928689:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537855.928690:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537855.928691:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537855.928692:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537855.928693:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19656 00000100:00080000:20.0:1584537855.928696:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537855.928707:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537855.928709:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537855.928709:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537855.928710:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537855.928711:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537855.928712:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537855.928713:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537855.928714:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537855.981569:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584537855.981575:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584537855.981576:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584537855.981577:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584537855.981579:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584537855.981581:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584537855.981582:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584537855.981584:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537937 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584537855.981613:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078435024 new x1661492078435056 00000800:00020000:4.0:1584537861.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537861.916081:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537861.916083:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537861.916620:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584537861.928701:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537855/real 1584537855] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537942 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584537861.928711:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584537861.928712:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584537861.928714:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584537861.928730:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537861.928732:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537861.928737:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537861.928738:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537861.928739:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537861.928740:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584537861.928742:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584537861.928744:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19661 00000100:00080000:20.0:1584537861.928747:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584537861.928753:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537861.928754:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537861.928755:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537861.928756:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537861.928757:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537861.928758:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584537861.928759:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584537861.928760:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537861.973818:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584537861.973824:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584537861.973826:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584537861.973828:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584537861.973830:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584537861.973832:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584537861.973834:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584537861.973836:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537942 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:0.0:1584537861.973846:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078435056 new x1661492078435136 00000800:00020000:6.0:1584537867.916104:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584537867.916107:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584537867.916109:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537867.916520:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584537867.928602:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537861/real 1584537861] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537948 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584537867.928611:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584537867.928613:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584537867.928614:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584537867.928620:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584537867.928623:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537867.928623:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584537867.928625:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537867.928625:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584537867.928627:0:4532:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584537867.928629:0:4532:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584537867.928630:0:4532:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19667 00000100:00080000:2.0:1584537867.928632:0:4532:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584537867.928645:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584537867.928646:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537867.928647:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584537867.928648:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537867.928649:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584537867.928650:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537867.928651:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584537867.928652:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537867.988470:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584537867.988476:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584537867.988478:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584537867.988480:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584537867.988481:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584537867.988483:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584537867.988485:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584537867.988487:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537948 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584537867.988496:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078435136 new x1661492078435168 00000800:00000100:0.0:1584537874.140967:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537874.141554:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584537874.153607:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537867/real 1584537867] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537954 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584537874.153615:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584537874.153617:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584537874.153618:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584537874.153623:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537874.153624:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537874.153625:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537874.153626:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537874.153627:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537874.153629:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584537874.153629:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584537874.153631:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19673 00000100:00080000:4.0:1584537874.153633:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584537874.153643:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537874.153644:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537874.153645:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537874.153646:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537874.153647:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537874.153648:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537874.153650:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537874.153651:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537874.219608:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584537874.219613:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584537874.219615:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584537874.219616:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584537874.219617:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584537874.219619:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584537874.219621:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584537874.219623:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537954 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584537874.219630:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078435168 new x1661492078435216 00000800:00020000:2.0:1584537879.916108:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537879.916110:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537879.916112:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537879.916532:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584537879.928615:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537874/real 1584537874] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537961 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584537879.928623:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584537879.928624:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584537879.928625:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584537879.928631:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537879.928632:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537879.928633:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537879.928634:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537879.928635:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537879.928636:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584537879.928636:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584537879.928638:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19680 00000100:00080000:4.0:1584537879.928640:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584537879.928649:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537879.928650:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537879.928650:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537879.928652:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537879.928653:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537879.928654:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584537879.928655:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584537879.928656:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537879.981625:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584537879.981630:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584537879.981631:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584537879.981632:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584537879.981634:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584537879.981636:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584537879.981637:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584537879.981639:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537961 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584537879.981669:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078435216 new x1661492078435248 00000800:00020000:4.0:1584537885.916041:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537885.916043:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537885.916045:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537885.916490:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537885.928589:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537879/real 1584537879] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537966 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537885.928597:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537885.928598:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537885.928599:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537885.928614:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537885.928616:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537885.928621:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537885.928623:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537885.928624:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537885.928626:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537885.928626:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537885.928628:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19685 00000100:00080000:8.0:1584537885.928630:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537885.928635:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537885.928636:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537885.928636:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537885.928638:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537885.928638:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537885.928639:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537885.928640:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537885.928641:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537885.987634:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584537885.987638:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584537885.987639:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584537885.987640:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584537885.987641:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584537885.987643:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584537885.987644:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584537885.987646:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537966 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584537885.987655:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078435248 new x1661492078435312 00000800:00020000:4.0:1584537891.916114:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537891.916117:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537891.916118:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537891.916558:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537891.928634:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537885/real 1584537885] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537972 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537891.928642:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537891.928643:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537891.928644:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537891.928650:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537891.928651:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537891.928653:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537891.928654:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537891.928655:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537891.928656:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537891.928657:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537891.928658:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19691 00000100:00080000:8.0:1584537891.928660:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537891.928669:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537891.928670:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537891.928670:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537891.928672:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537891.928672:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537891.928673:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537891.928674:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537891.928675:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537891.980364:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584537891.980369:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584537891.980370:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584537891.980371:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584537891.980372:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584537891.980374:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584537891.980375:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584537891.980377:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537972 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584537891.980387:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078435312 new x1661492078435360 00000800:00000100:0.0:1584537898.141671:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537898.142096:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537898.154145:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537891/real 1584537891] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537978 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537898.154153:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537898.154154:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537898.154155:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537898.154163:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537898.154164:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537898.154165:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537898.154166:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537898.154167:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537898.154168:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537898.154169:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537898.154170:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19697 00000100:00080000:8.0:1584537898.154172:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537898.154180:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537898.154182:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537898.154182:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537898.154183:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537898.154184:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537898.154185:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537898.154186:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537898.154187:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537898.212311:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584537898.212316:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584537898.212317:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584537898.212318:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584537898.212320:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584537898.212322:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584537898.212324:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584537898.212325:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537978 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584537898.212355:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078435360 new x1661492078435392 00000800:00020000:2.0:1584537903.916136:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537903.916139:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537903.916141:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537903.916530:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537903.928581:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537898/real 1584537898] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537985 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537903.928589:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537903.928591:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537903.928592:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537903.928597:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537903.928599:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537903.928600:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537903.928601:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537903.928602:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537903.928603:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537903.928604:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537903.928606:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19704 00000100:00080000:8.0:1584537903.928608:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537903.928616:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537903.928617:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537903.928618:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537903.928619:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537903.928620:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537903.928621:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537903.928622:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537903.928623:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584537903.982733:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584537903.982737:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584537903.982739:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584537903.982740:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584537903.982741:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584537903.982743:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584537903.982745:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584537903.982746:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537985 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584537903.982776:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078435392 new x1661492078435440 00000800:00020000:2.0:1584537909.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537909.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537909.916086:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537909.916609:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537909.928748:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537903/real 1584537903] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537990 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537909.928758:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537909.928759:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537909.928761:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537909.928780:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537909.928782:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537909.928787:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537909.928788:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537909.928789:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537909.928791:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537909.928792:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537909.928793:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19709 00000100:00080000:8.0:1584537909.928796:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537909.928803:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537909.928804:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537909.928805:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537909.928806:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537909.928807:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537909.928808:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537909.928809:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537909.928810:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537909.975358:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537909.975364:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537909.975366:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537909.975367:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537909.975369:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537909.975371:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537909.975373:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537909.975375:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537990 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584537909.975384:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078435440 new x1661492078435504 00000800:00020000:2.0:1584537915.916044:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537915.916047:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537915.916049:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537915.916438:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537915.928489:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537909/real 1584537909] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537996 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537915.928499:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537915.928500:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537915.928502:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537915.928509:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537915.928511:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537915.928512:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537915.928513:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537915.928514:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537915.928516:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537915.928517:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537915.928519:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19715 00000100:00080000:8.0:1584537915.928522:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537915.928537:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537915.928538:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537915.928539:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537915.928540:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537915.928541:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537915.928542:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537915.928544:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537915.928545:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537915.987582:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537915.987589:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537915.987592:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537915.987593:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537915.987594:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537915.987597:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537915.987599:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537915.987601:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584537996 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584537915.987630:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078435504 new x1661492078435536 00000800:00000100:0.0:1584537922.140857:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537922.141415:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537922.153492:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537915/real 1584537915] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538002 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537922.153501:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537922.153502:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537922.153503:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537922.153510:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537922.153512:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537922.153513:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537922.153514:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537922.153515:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537922.153516:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537922.153517:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537922.153519:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19721 00000100:00080000:8.0:1584537922.153521:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537922.153531:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537922.153532:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537922.153533:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537922.153534:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537922.153536:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537922.153537:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537922.153538:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537922.153539:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537922.199624:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537922.199629:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537922.199631:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537922.199632:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537922.199634:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537922.199636:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537922.199638:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537922.199640:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538002 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584537922.199669:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078435536 new x1661492078435584 00000800:00020000:2.0:1584537927.916058:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537927.916062:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537927.916064:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537927.916453:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537927.928529:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537922/real 1584537922] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538009 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537927.928537:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537927.928538:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537927.928539:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537927.928546:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537927.928548:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537927.928548:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537927.928550:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537927.928550:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537927.928552:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537927.928553:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537927.928554:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19728 00000100:00080000:8.0:1584537927.928556:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537927.928567:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537927.928569:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537927.928570:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537927.928572:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537927.928573:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537927.928574:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537927.928574:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537927.928576:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537927.993590:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537927.993596:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537927.993598:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537927.993600:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537927.993601:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537927.993604:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537927.993605:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537927.993607:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538009 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584537927.993640:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078435584 new x1661492078435616 00000800:00020000:4.0:1584537933.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537933.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537933.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537933.916552:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537933.928606:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537927/real 1584537927] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538014 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537933.928614:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537933.928616:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537933.928617:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537933.928631:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537933.928632:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537933.928638:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537933.928639:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537933.928640:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537933.928641:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537933.928642:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537933.928643:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19733 00000100:00080000:8.0:1584537933.928646:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537933.928653:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537933.928654:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537933.928655:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537933.928656:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537933.928657:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537933.928658:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537933.928659:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537933.928660:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537933.988826:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537933.988831:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537933.988833:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537933.988834:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537933.988835:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537933.988837:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537933.988839:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537933.988840:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538014 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584537933.988888:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078435616 new x1661492078435696 00000800:00020000:2.0:1584537939.916090:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537939.916092:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537939.916094:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537939.916535:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537939.928611:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537933/real 1584537933] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538020 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537939.928619:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537939.928620:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537939.928621:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537939.928627:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537939.928629:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537939.928629:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537939.928631:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537939.928631:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537939.928633:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537939.928633:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537939.928636:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19739 00000100:00080000:8.0:1584537939.928638:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537939.928646:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537939.928647:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537939.928648:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537939.928649:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537939.928649:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537939.928650:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537939.928651:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537939.928652:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537939.989679:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537939.989683:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537939.989685:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537939.989686:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537939.989687:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537939.989689:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537939.989691:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537939.989693:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538020 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584537939.989722:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078435696 new x1661492078435728 00000800:00000100:0.0:1584537946.141405:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537946.141817:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537946.153884:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537939/real 1584537939] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538026 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537946.153891:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537946.153892:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537946.153893:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537946.153899:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537946.153901:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537946.153901:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537946.153902:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537946.153903:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537946.153905:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537946.153906:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537946.153907:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19745 00000100:00080000:8.0:1584537946.153909:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537946.153917:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537946.153919:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537946.153919:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537946.153920:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537946.153921:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537946.153922:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537946.153923:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537946.153924:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537946.207458:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537946.207462:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537946.207463:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537946.207464:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537946.207465:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537946.207467:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537946.207468:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537946.207470:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538026 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584537946.207499:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078435728 new x1661492078435760 00000800:00020000:4.0:1584537951.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537951.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537951.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537951.916511:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537951.928598:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537946/real 1584537946] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538033 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537951.928606:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537951.928607:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537951.928609:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537951.928614:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537951.928616:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537951.928616:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537951.928619:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537951.928619:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537951.928621:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537951.928621:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537951.928623:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19752 00000100:00080000:8.0:1584537951.928625:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537951.928633:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537951.928635:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537951.928635:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537951.928636:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537951.928637:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537951.928638:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537951.928639:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537951.928640:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537951.985555:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537951.985560:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537951.985561:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537951.985562:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537951.985563:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537951.985565:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537951.985566:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537951.985568:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538033 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584537951.985600:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078435760 new x1661492078435808 00000800:00020000:2.0:1584537957.916057:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537957.916059:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537957.916062:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537957.916466:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537957.928547:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537951/real 1584537951] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538038 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537957.928555:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537957.928556:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537957.928557:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537957.928571:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537957.928574:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537957.928579:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537957.928580:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537957.928580:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537957.928582:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537957.928582:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537957.928584:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19757 00000100:00080000:8.0:1584537957.928586:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537957.928591:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537957.928592:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537957.928593:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537957.928594:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537957.928595:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537957.928596:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537957.928596:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537957.928598:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537958.000614:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537958.000618:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537958.000620:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537958.000621:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537958.000622:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537958.000624:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537958.000626:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537958.000627:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538038 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584537958.000636:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078435808 new x1661492078435872 00000800:00020000:2.0:1584537963.916128:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537963.916131:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537963.916133:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537963.916542:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537963.928623:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537957/real 1584537957] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538044 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537963.928631:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537963.928632:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537963.928634:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537963.928640:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537963.928642:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537963.928643:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537963.928644:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537963.928644:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537963.928646:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537963.928646:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537963.928648:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19763 00000100:00080000:8.0:1584537963.928650:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537963.928658:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537963.928659:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537963.928660:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537963.928661:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537963.928662:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537963.928663:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537963.928663:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537963.928664:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537963.986796:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537963.986801:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537963.986803:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537963.986803:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537963.986805:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537963.986807:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537963.986808:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537963.986810:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538044 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584537963.986839:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078435872 new x1661492078435920 00000800:00000100:0.0:1584537970.141632:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537970.142056:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537970.154103:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537963/real 1584537963] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538050 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537970.154115:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537970.154116:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537970.154118:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537970.154123:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537970.154124:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537970.154125:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537970.154126:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537970.154127:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537970.154128:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537970.154129:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537970.154130:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19769 00000100:00080000:8.0:1584537970.154132:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537970.154140:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537970.154141:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537970.154142:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537970.154143:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537970.154144:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537970.154145:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537970.154145:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537970.154146:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537970.201715:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537970.201720:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537970.201721:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537970.201722:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537970.201723:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537970.201725:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537970.201726:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537970.201728:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538050 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584537970.201762:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078435920 new x1661492078435952 00000800:00020000:2.0:1584537975.916049:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537975.916052:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537975.916054:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537975.916519:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537975.928570:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537970/real 1584537970] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538057 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537975.928578:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537975.928580:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537975.928581:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537975.928586:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537975.928587:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537975.928588:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537975.928589:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537975.928590:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537975.928591:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537975.928592:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537975.928593:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19776 00000100:00080000:8.0:1584537975.928595:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537975.928604:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537975.928605:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537975.928606:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537975.928607:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537975.928608:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537975.928609:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537975.928610:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537975.928611:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537975.980376:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537975.980381:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537975.980382:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537975.980383:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537975.980385:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537975.980387:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537975.980388:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537975.980390:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538057 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584537975.980399:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078435952 new x1661492078435984 00000800:00020000:2.0:1584537981.916086:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584537981.938902:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584537981.963075:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537981.963497:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584537981.975553:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537975/real 1584537976] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538062 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584537981.975563:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584537981.975565:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584537981.975567:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584537981.975588:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537981.975590:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537981.975595:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537981.975596:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537981.975597:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537981.975599:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584537981.975600:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584537981.975602:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19781 00000100:00080000:8.0:1584537981.975605:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584537981.975613:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537981.975615:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537981.975615:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537981.975617:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537981.975618:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537981.975619:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584537981.975620:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584537981.975621:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537982.037848:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537982.037856:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537982.037858:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537982.037860:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537982.037861:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537982.037864:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537982.037866:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537982.037868:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538062 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584537982.037898:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078435984 new x1661492078436064 00000800:00020000:4.0:1584537987.916172:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537987.916175:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537987.916177:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537987.916590:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584537987.928642:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537982/real 1584537982] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538069 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584537987.928650:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584537987.928651:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584537987.928652:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584537987.928658:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537987.928660:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537987.928661:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537987.928663:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537987.928663:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537987.928665:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584537987.928665:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584537987.928667:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19787 00000100:00080000:10.0:1584537987.928669:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584537987.928679:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537987.928681:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537987.928681:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537987.928682:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537987.928683:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537987.928684:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537987.928685:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537987.928686:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537987.977901:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537987.977906:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537987.977908:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537987.977909:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537987.977910:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537987.977913:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537987.977914:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537987.977916:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538069 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584537987.977945:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078436064 new x1661492078436096 00000800:00000100:0.0:1584537994.141055:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584537994.141548:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584537994.153600:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537987/real 1584537987] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538074 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584537994.153609:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584537994.153610:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584537994.153611:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584537994.153616:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537994.153618:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537994.153619:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537994.153620:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537994.153621:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537994.153622:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584537994.153623:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584537994.153624:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19793 00000100:00080000:10.0:1584537994.153626:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584537994.153635:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537994.153636:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537994.153637:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537994.153638:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537994.153639:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537994.153640:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537994.153641:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537994.153642:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537994.192921:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537994.192926:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537994.192928:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537994.192929:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537994.192930:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537994.192932:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537994.192934:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537994.192936:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538074 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584537994.192964:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078436096 new x1661492078436144 00000800:00020000:4.0:1584537999.916110:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584537999.916112:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584537999.916114:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584537999.916554:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584537999.928606:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537994/real 1584537994] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538081 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584537999.928613:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584537999.928614:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584537999.928616:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584537999.928621:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537999.928622:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537999.928623:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537999.928624:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537999.928625:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537999.928626:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584537999.928627:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584537999.928628:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19800 00000100:00080000:10.0:1584537999.928630:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584537999.928638:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537999.928639:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537999.928640:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537999.928641:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537999.928642:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537999.928643:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584537999.928644:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584537999.928645:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584537999.965390:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584537999.965394:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584537999.965395:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584537999.965396:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584537999.965397:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584537999.965399:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584537999.965400:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584537999.965402:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538081 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584537999.965432:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078436144 new x1661492078436176 00000800:00020000:2.0:1584538005.916050:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538005.916053:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538005.916055:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538005.916526:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584538005.928575:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584537999/real 1584537999] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538086 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584538005.928587:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584538005.928588:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584538005.928589:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584538005.928602:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538005.928603:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538005.928609:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538005.928610:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538005.928611:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538005.928612:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584538005.928612:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584538005.928614:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19805 00000100:00080000:10.0:1584538005.928616:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584538005.928621:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538005.928622:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538005.928623:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538005.928624:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538005.928625:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538005.928626:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538005.928626:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538005.928628:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538005.936501:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584538005.936505:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584538005.936507:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584538005.936508:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584538005.936509:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584538005.936511:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584538005.936512:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584538005.936514:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538086 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538005.936522:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078436176 new x1661492078436240 00000800:00020000:4.0:1584538011.916051:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584538011.916054:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584538011.916056:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538011.916489:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538011.928544:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538005/real 1584538005] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538092 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538011.928557:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538011.928558:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538011.928559:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538011.928565:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538011.928567:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538011.928568:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538011.928569:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538011.928570:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538011.928571:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538011.928572:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538011.928574:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19811 00000100:00080000:14.0:1584538011.928576:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538011.928585:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538011.928587:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538011.928587:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538011.928588:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538011.928589:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538011.928590:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538011.928591:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538011.928592:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538011.936501:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584538011.936506:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584538011.936507:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584538011.936508:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584538011.936509:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584538011.936512:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584538011.936513:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584538011.936515:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538092 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538011.936523:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078436240 new x1661492078436288 00000800:00000100:0.0:1584538018.141293:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538018.141725:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538018.153775:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538011/real 1584538011] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538098 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538018.153783:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538018.153784:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538018.153785:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538018.153791:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538018.153793:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538018.153793:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538018.153795:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538018.153795:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538018.153797:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538018.153797:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538018.153799:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19817 00000100:00080000:14.0:1584538018.153801:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538018.153810:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538018.153811:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538018.153812:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538018.153813:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538018.153814:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538018.153815:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538018.153817:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538018.153818:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538018.161554:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584538018.161558:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584538018.161559:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584538018.161560:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584538018.161562:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584538018.161563:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584538018.161565:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584538018.161567:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538098 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538018.161596:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078436288 new x1661492078436320 00000800:00020000:4.0:1584538023.916071:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584538023.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584538023.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538023.916525:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538023.928651:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538018/real 1584538018] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538105 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538023.928662:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538023.928663:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538023.928665:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538023.928672:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538023.928674:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538023.928676:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538023.928677:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538023.928678:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538023.928680:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538023.928681:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538023.928683:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19824 00000100:00080000:14.0:1584538023.928686:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538023.928699:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538023.928700:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538023.928701:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538023.928702:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538023.928704:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538023.928705:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538023.928705:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538023.928707:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538023.936638:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584538023.936644:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584538023.936646:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584538023.936647:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584538023.936649:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584538023.936651:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584538023.936653:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584538023.936655:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538105 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538023.936683:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078436320 new x1661492078436368 00000800:00020000:2.0:1584538029.916057:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538029.916059:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538029.916062:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538029.916479:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538029.928526:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538023/real 1584538023] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538110 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538029.928535:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538029.928537:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538029.928538:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538029.928557:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538029.928559:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538029.928565:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538029.928566:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538029.928567:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538029.928569:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538029.928570:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538029.928572:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19829 00000100:00080000:14.0:1584538029.928574:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538029.928581:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538029.928582:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538029.928584:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538029.928585:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538029.928586:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538029.928587:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538029.928588:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538029.928589:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538029.936673:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584538029.936678:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584538029.936680:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584538029.936681:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584538029.936683:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584538029.936685:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584538029.936686:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584538029.936688:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538110 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538029.936717:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078436368 new x1661492078436432 00000800:00020000:4.0:1584538035.916058:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584538035.916061:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584538035.916063:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538035.916479:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538035.928529:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538029/real 1584538029] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538116 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538035.928537:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538035.928539:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538035.928540:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538035.928546:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538035.928548:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538035.928549:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538035.928550:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538035.928551:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538035.928552:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538035.928553:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538035.928555:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19835 00000100:00080000:14.0:1584538035.928557:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538035.928570:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538035.928571:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538035.928572:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538035.928573:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538035.928574:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538035.928575:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538035.928575:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538035.928577:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538035.936537:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584538035.936541:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584538035.936543:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584538035.936544:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584538035.936545:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584538035.936548:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584538035.936549:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584538035.936551:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538116 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538035.936583:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078436432 new x1661492078436464 00000800:00000100:0.0:1584538042.140891:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538042.141469:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538042.153528:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538035/real 1584538035] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538122 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538042.153539:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538042.153541:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538042.153543:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538042.153551:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538042.153553:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538042.153554:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538042.153555:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538042.153556:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538042.153557:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538042.153558:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538042.153561:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19841 00000100:00080000:14.0:1584538042.153565:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538042.153581:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538042.153582:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538042.153583:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538042.153584:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538042.153585:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538042.153586:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538042.153586:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538042.153588:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538042.161683:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584538042.161690:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584538042.161692:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584538042.161693:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584538042.161695:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584538042.161697:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584538042.161699:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584538042.161701:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538122 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538042.161730:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078436464 new x1661492078436512 00000800:00020000:4.0:1584538047.916143:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584538047.916146:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584538047.916148:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538047.916561:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538047.928617:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538042/real 1584538042] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538129 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538047.928625:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538047.928627:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538047.928628:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538047.928635:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538047.928637:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538047.928638:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538047.928639:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538047.928640:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538047.928642:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538047.928643:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538047.928645:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19848 00000100:00080000:14.0:1584538047.928648:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538047.928661:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538047.928662:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538047.928663:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538047.928664:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538047.928665:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538047.928666:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538047.928667:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538047.928668:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584538047.936392:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584538047.936397:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584538047.936398:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584538047.936399:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584538047.936401:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584538047.936403:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584538047.936404:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584538047.936406:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538129 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538047.936436:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078436512 new x1661492078436544 00000800:00020000:2.0:1584538053.916096:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538053.916099:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538053.916101:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538053.916484:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538053.928538:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538047/real 1584538047] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538134 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538053.928547:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538053.928548:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538053.928550:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538053.928565:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538053.928567:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538053.928573:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538053.928576:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538053.928576:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538053.928578:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538053.928579:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538053.928580:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19853 00000100:00080000:14.0:1584538053.928583:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538053.928589:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538053.928591:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538053.928592:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538053.928593:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538053.928594:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538053.928595:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538053.928596:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538053.928597:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538053.936450:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584538053.936454:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584538053.936456:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584538053.936457:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584538053.936458:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584538053.936460:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584538053.936461:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584538053.936463:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538134 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538053.936492:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078436544 new x1661492078436624 00000800:00020000:2.0:1584538059.916066:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538059.916069:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (3): c: 6, oc: 0, rc: 8 00000800:00000100:2.0:1584538059.916071:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538059.916482:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538059.928538:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538053/real 1584538053] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538140 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538059.928545:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538059.928546:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538059.928548:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538059.928553:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538059.928557:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538059.928558:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538059.928559:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538059.928560:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538059.928561:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538059.928562:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538059.928563:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19859 00000100:00080000:14.0:1584538059.928565:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538059.928573:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538059.928575:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538059.928575:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538059.928576:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538059.928577:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538059.928578:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538059.928579:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538059.928580:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538059.936374:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584538059.936378:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584538059.936379:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584538059.936380:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584538059.936382:0:25724:0:(client.c:2816:ptlrpc_resend_req()) @@@ going to resend req@ffff8d2babe46780 x1661492078436640/t0(0) o17->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 456/432 e 0 to 0 dl 1584538150 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000100:00080000:8.0:1584538059.936388:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584538059.936390:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584538059.936391:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584538059.936393:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538140 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538059.936422:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078436624 new x1661492078436688 00000800:00000100:0.0:1584538066.141311:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538066.141727:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538066.153776:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538059/real 1584538059] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538146 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538066.153783:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538066.153784:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538066.153787:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538066.153792:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538066.153794:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538066.153795:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538066.153796:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538066.153797:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538066.153798:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538066.153799:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538066.153800:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19865 00000100:00080000:14.0:1584538066.153802:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538066.153810:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538066.153812:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538066.153812:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538066.153813:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538066.153814:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538066.153815:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538066.153816:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538066.153817:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538066.161761:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584538066.161766:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584538066.161767:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584538066.161768:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584538066.161769:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584538066.161771:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584538066.161772:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584538066.161774:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538146 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538066.161803:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078436688 new x1661492078436720 00000800:00020000:4.0:1584538071.916096:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584538071.916099:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584538071.916101:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538071.916567:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538071.928644:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538066/real 1584538066] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538153 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538071.928656:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538071.928657:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538071.928659:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538071.928665:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538071.928667:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538071.928667:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538071.928668:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538071.928669:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538071.928670:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538071.928671:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538071.928673:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19872 00000100:00080000:14.0:1584538071.928675:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538071.928683:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538071.928684:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538071.928685:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538071.928686:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538071.928687:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538071.928688:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538071.928688:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538071.928690:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538071.936341:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584538071.936345:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584538071.936346:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584538071.936347:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584538071.936348:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584538071.936350:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584538071.936352:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584538071.936353:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538153 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538071.936382:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078436720 new x1661492078436768 00000800:00020000:2.0:1584538077.916114:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538077.916117:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538077.916119:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538077.916535:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538077.928582:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538071/real 1584538071] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538158 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538077.928590:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538077.928591:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538077.928592:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538077.928606:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538077.928608:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538077.928613:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538077.928614:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538077.928615:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538077.928616:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538077.928617:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538077.928618:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19877 00000100:00080000:14.0:1584538077.928620:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538077.928626:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538077.928627:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538077.928628:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538077.928629:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538077.928629:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538077.928631:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538077.928632:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538077.928634:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538077.936459:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584538077.936463:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584538077.936465:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584538077.936465:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584538077.936467:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584538077.936468:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584538077.936470:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584538077.936472:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538158 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538077.936501:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078436768 new x1661492078436832 00000800:00020000:2.0:1584538083.916113:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538083.916116:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538083.916118:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538083.916563:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538083.928608:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538077/real 1584538077] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538164 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538083.928616:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538083.928617:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538083.928619:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538083.928626:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538083.928627:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538083.928628:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538083.928629:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538083.928630:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538083.928631:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538083.928632:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538083.928634:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19883 00000100:00080000:14.0:1584538083.928636:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538083.928645:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538083.928647:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538083.928647:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538083.928648:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538083.928650:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538083.928651:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538083.928652:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538083.928653:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538083.936626:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584538083.936632:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584538083.936634:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584538083.936635:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584538083.936636:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584538083.936638:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584538083.936640:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584538083.936642:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538164 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538083.936670:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078436832 new x1661492078436880 00000800:00020000:2.0:1584538089.916043:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538089.916045:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538089.916047:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538089.916489:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538089.928538:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538083/real 1584538083] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538170 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538089.928547:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538089.928548:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538089.928549:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538089.928555:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538089.928557:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538089.928558:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538089.928559:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538089.928560:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538089.928561:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538089.928562:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538089.928564:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19889 00000100:00080000:14.0:1584538089.928566:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538089.928577:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538089.928578:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538089.928580:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538089.928581:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538089.928582:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538089.928583:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538089.928584:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538089.928585:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538089.936587:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584538089.936593:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584538089.936594:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584538089.936595:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584538089.936597:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584538089.936599:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584538089.936601:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584538089.936602:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538170 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538089.936631:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078436880 new x1661492078436912 00000800:00020000:0.0:1584538095.916069:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584538095.916072:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584538095.916074:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538095.916484:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538095.928541:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538089/real 1584538089] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538176 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538095.928549:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538095.928550:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538095.928551:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538095.928558:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538095.928560:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538095.928560:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538095.928562:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538095.928562:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538095.928564:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538095.928564:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538095.928566:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19895 00000100:00080000:14.0:1584538095.928568:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538095.928580:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538095.928581:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538095.928582:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538095.928583:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538095.928584:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538095.928585:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538095.928586:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538095.928587:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538095.936782:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584538095.936788:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584538095.936789:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584538095.936790:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584538095.936792:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584538095.936794:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584538095.936795:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584538095.936797:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538176 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538095.936826:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078436912 new x1661492078436944 00000800:00020000:2.0:1584538101.916128:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538101.916131:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538101.916133:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538101.916635:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538101.928709:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538095/real 1584538095] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538182 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538101.928719:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538101.928720:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538101.928722:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538101.928740:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538101.928742:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538101.928747:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538101.928749:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538101.928749:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538101.928751:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538101.928752:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538101.928754:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19901 00000100:00080000:14.0:1584538101.928757:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538101.928764:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538101.928765:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538101.928766:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538101.928768:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538101.928768:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538101.928770:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538101.928770:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538101.928771:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538101.936670:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584538101.936677:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584538101.936679:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584538101.936680:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584538101.936681:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584538101.936684:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584538101.936685:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584538101.936687:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538182 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584538101.936717:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078436944 new x1661492078437024 00000800:00020000:2.0:1584538107.916068:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538107.916071:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538107.916072:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538107.916539:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584538107.928593:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538101/real 1584538101] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538188 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584538107.928601:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584538107.928602:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584538107.928604:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584538107.928611:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538107.928613:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538107.928614:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538107.928615:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538107.928616:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538107.928617:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584538107.928617:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584538107.928619:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19907 00000100:00080000:16.0:1584538107.928621:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584538107.928630:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538107.928632:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538107.928632:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538107.928633:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538107.928634:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538107.928635:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538107.928636:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538107.928637:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538107.936608:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584538107.936612:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584538107.936613:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584538107.936614:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584538107.936615:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584538107.936617:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584538107.936618:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584538107.936620:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538188 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584538107.936650:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078437024 new x1661492078437056 00000800:00020000:8.0:1584538113.916136:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584538113.916139:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584538113.916140:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538113.916553:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584538113.928632:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538107/real 1584538107] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538194 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584538113.928641:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584538113.928645:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584538113.928646:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584538113.928652:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538113.928654:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538113.928655:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538113.928656:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538113.928657:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538113.928658:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584538113.928659:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584538113.928660:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19913 00000100:00080000:16.0:1584538113.928663:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584538113.928672:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538113.928673:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538113.928674:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538113.928675:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538113.928676:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538113.928677:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538113.928678:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538113.928679:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538113.936634:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584538113.936639:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584538113.936641:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584538113.936641:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584538113.936643:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584538113.936645:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584538113.936646:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584538113.936649:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538194 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584538113.936678:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078437056 new x1661492078437104 00000800:00020000:4.0:1584538119.916089:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584538119.916092:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584538119.916094:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538119.916531:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584538119.928584:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538113/real 1584538113] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538200 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584538119.928593:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584538119.928594:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584538119.928596:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584538119.928601:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538119.928603:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538119.928604:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538119.928605:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538119.928606:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538119.928607:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584538119.928608:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584538119.928609:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19919 00000100:00080000:16.0:1584538119.928611:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584538119.928620:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538119.928621:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538119.928622:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538119.928623:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538119.928623:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538119.928624:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538119.928625:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538119.928626:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538119.936375:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584538119.936378:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584538119.936380:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584538119.936381:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584538119.936382:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584538119.936384:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584538119.936385:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584538119.936387:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538200 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584538119.936418:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078437104 new x1661492078437136 00000800:00020000:2.0:1584538125.916073:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538125.916075:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538125.916077:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538125.916458:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584538125.928509:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538119/real 1584538119] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538206 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584538125.928517:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584538125.928518:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584538125.928520:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584538125.928533:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538125.928535:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538125.928541:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538125.928542:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538125.928543:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538125.928544:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584538125.928545:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584538125.928546:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19925 00000100:00080000:16.0:1584538125.928548:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584538125.928553:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538125.928555:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538125.928555:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538125.928556:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538125.928557:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538125.928558:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538125.928559:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538125.928561:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538125.936726:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584538125.936730:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584538125.936731:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584538125.936732:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584538125.936733:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584538125.936735:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584538125.936736:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584538125.936738:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538206 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538125.936768:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078437136 new x1661492078437200 00000800:00020000:2.0:1584538131.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538131.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538131.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538131.916519:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538131.928579:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538125/real 1584538125] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538212 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538131.928587:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538131.928588:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538131.928589:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538131.928596:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538131.928598:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538131.928598:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538131.928602:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538131.928602:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538131.928603:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538131.928604:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538131.928606:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19931 00000100:00080000:18.0:1584538131.928608:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538131.928618:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538131.928619:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538131.928620:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538131.928621:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538131.928622:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538131.928623:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538131.928623:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538131.928625:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538131.936819:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584538131.936823:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584538131.936825:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584538131.936825:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584538131.936827:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584538131.936829:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584538131.936831:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584538131.936833:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538212 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538131.936861:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078437200 new x1661492078437248 00000800:00020000:2.0:1584538137.916127:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538137.916130:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538137.916132:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538137.916574:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538137.928626:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538131/real 1584538131] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538218 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538137.928634:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538137.928635:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538137.928636:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538137.928642:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538137.928645:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538137.928645:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538137.928647:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538137.928647:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538137.928649:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538137.928649:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538137.928651:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19937 00000100:00080000:18.0:1584538137.928653:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538137.928661:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538137.928663:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538137.928663:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538137.928665:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538137.928665:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538137.928666:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538137.928667:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538137.928668:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538137.936720:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584538137.936724:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584538137.936726:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584538137.936726:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584538137.936728:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584538137.936730:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584538137.936731:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584538137.936733:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538218 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538137.936763:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078437248 new x1661492078437280 00000800:00020000:4.0:1584538143.916102:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584538143.916105:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584538143.916108:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538143.916593:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538143.928690:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538137/real 1584538137] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538224 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538143.928711:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538143.928712:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538143.928716:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538143.928724:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538143.928726:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538143.928726:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538143.928728:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538143.928728:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538143.928730:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538143.928731:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538143.928733:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19943 00000100:00080000:18.0:1584538143.928736:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538143.928751:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538143.928752:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538143.928753:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538143.928754:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538143.928755:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538143.928756:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538143.928756:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538143.928769:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538143.937416:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584538143.937436:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584538143.937438:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584538143.937439:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584538143.937441:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584538143.937443:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584538143.937445:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584538143.937447:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538224 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538143.937480:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078437280 new x1661492078437328 00000800:00020000:2.0:1584538149.916073:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538149.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538149.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538149.916554:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538149.928623:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538143/real 1584538143] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538230 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538149.928638:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538149.928640:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538149.928642:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538149.928664:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538149.928666:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538149.928672:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538149.928674:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538149.928674:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538149.928676:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538149.928677:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538149.928679:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19949 00000100:00080000:18.0:1584538149.928683:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538149.928691:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538149.928692:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538149.928693:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538149.928694:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538149.928695:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538149.928696:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538149.928697:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538149.928698:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538149.937092:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584538149.937099:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584538149.937102:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584538149.937103:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584538149.937105:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584538149.937108:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584538149.937109:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584538149.937111:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538230 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538149.937146:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078437328 new x1661492078437392 00000100:00000400:16.0:1584538150.601062:0:25735:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584538056/real 1584538056] req@ffff8d2babe46c00 x1661492078436656/t0(0) o17->public1-OST0001-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 456/432 e 0 to 1 dl 1584538150 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 00000100:02000400:16.0:1584538150.601071:0:25735:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection to public1-OST0001 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584538150.601072:0:25735:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584538150.601074:0:25735:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0001_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584538150.601080:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538150.601082:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538150.601083:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538150.601084:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:16.0:1584538150.601085:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584538150.601087:0:4060:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 18729 00000100:00080000:16.0:1584538150.601089:0:4060:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584538150.601100:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538150.601101:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538150.601102:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538150.601104:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538150.601104:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538150.601105:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538150.601106:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538150.601108:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538150.601109:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584538150.601110:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000800:00020000:4.0:1584538155.916137:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584538155.916141:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 7, oc: 0, rc: 8 00000800:00000100:4.0:1584538155.916143:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538155.916590:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538155.928646:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538149/real 1584538149] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538236 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538155.928656:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538155.928658:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538155.928659:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538155.928667:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538155.928669:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538155.928670:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538155.928671:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538155.928672:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538155.928674:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538155.928675:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538155.928677:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19955 00000100:00080000:18.0:1584538155.928679:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538155.928692:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538155.928694:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538155.928694:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538155.928695:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538155.928696:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538155.928697:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538155.928699:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538155.928701:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538155.936701:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584538155.936708:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584538155.936710:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584538155.936711:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584538155.936713:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584538155.936716:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584538155.936717:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584538155.936719:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538236 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538155.936749:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078437392 new x1661492078437440 00000100:00080000:10.0:1584538156.928042:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:10.0:1584538156.928044:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0001_UUID on 10.10.2.22@o2ib failed (-110) 00000800:00020000:0.0:1584538161.916059:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584538161.916063:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584538161.916065:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538161.916480:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538161.928521:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538155/real 1584538155] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538242 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538161.928532:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538161.928534:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538161.928536:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538161.928543:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538161.928545:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538161.928546:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538161.928547:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538161.928548:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538161.928550:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538161.928551:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538161.928553:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19961 00000100:00080000:18.0:1584538161.928556:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538161.928571:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538161.928573:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538161.928573:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538161.928575:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538161.928577:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538161.928578:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538161.928579:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538161.928580:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538161.936749:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584538161.936756:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584538161.936758:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584538161.936759:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584538161.936762:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584538161.936764:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584538161.936766:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584538161.936768:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538242 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538161.936796:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078437440 new x1661492078437488 00000800:00020000:0.0:1584538167.916060:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584538167.916063:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584538167.916065:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538167.916453:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538167.928506:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538161/real 1584538161] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538248 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538167.928515:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538167.928516:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538167.928518:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538167.928525:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538167.928527:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538167.928528:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538167.928529:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538167.928530:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538167.928532:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538167.928532:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538167.928534:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19967 00000100:00080000:18.0:1584538167.928537:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538167.928549:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538167.928551:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538167.928552:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538167.928554:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538167.928555:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538167.928556:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538167.928556:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538167.928558:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538167.936506:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584538167.936512:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584538167.936514:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584538167.936515:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584538167.936517:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584538167.936519:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584538167.936522:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584538167.936523:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538248 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538167.936552:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078437488 new x1661492078437520 00000800:00020000:6.0:1584538173.916169:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584538173.916172:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584538173.916174:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538173.916644:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538173.928698:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538167/real 1584538167] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538254 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538173.928707:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538173.928708:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538173.928709:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538173.928725:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538173.928727:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538173.928728:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:18.0:1584538173.928730:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:18.0:1584538173.928731:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538173.928733:0:4066:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19956 00000100:00080000:18.0:1584538173.928735:0:4066:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 17522 00000100:00080000:18.0:1584538173.928737:0:4066:0:(import.c:596:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0001 (at 10.10.2.21@o2ib) 00000100:00080000:18.0:1584538173.928738:0:4066:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.21@o2ib/10.10.2.21@o2ib 00000100:00080000:18.0:1584538173.928746:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538173.928747:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538173.928748:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538173.928749:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538173.928750:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538173.928751:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19973 00000100:00080000:18.0:1584538173.928752:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538173.928756:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538173.928757:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538173.928758:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538173.928759:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538173.928760:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538173.928761:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538173.928762:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538173.928763:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538173.928953:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:10.0:1584538173.928955:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0001_UUID on 10.10.2.21@o2ib failed (-19) 00000100:00080000:10.0:1584538173.937123:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584538173.937133:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584538173.937136:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584538173.937138:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584538173.937141:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584538173.937146:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584538173.937150:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584538173.937154:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538254 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538173.937200:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078437520 new x1661492078437600 00000800:00020000:2.0:1584538179.916060:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538179.916063:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538179.916065:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538179.916447:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538179.928497:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538173/real 1584538173] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538260 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538179.928505:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538179.928506:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538179.928507:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538179.928514:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538179.928516:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538179.928516:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538179.928518:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538179.928518:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538179.928520:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538179.928520:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538179.928522:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19979 00000100:00080000:18.0:1584538179.928524:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538179.928532:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538179.928533:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538179.928534:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538179.928535:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538179.928536:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538179.928537:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538179.928538:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538179.928539:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538179.936458:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584538179.936462:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584538179.936464:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584538179.936464:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584538179.936466:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584538179.936468:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584538179.936469:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584538179.936471:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538260 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538179.936499:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078437600 new x1661492078437632 00000800:00020000:2.0:1584538185.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538185.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538185.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538185.916524:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538185.928576:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538179/real 1584538179] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538266 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538185.928584:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538185.928588:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538185.928589:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538185.928595:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538185.928597:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538185.928597:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538185.928599:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538185.928599:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538185.928601:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538185.928601:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538185.928603:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19985 00000100:00080000:18.0:1584538185.928605:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538185.928613:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538185.928614:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538185.928615:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538185.928616:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538185.928616:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538185.928618:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538185.928618:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538185.928619:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538185.976559:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584538185.976563:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584538185.976564:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584538185.976565:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584538185.976567:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584538185.976569:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584538185.976570:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584538185.976572:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538266 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538185.976601:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078437632 new x1661492078437664 00000800:00000100:0.0:1584538192.141204:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538192.141756:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538192.153807:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538185/real 1584538185] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538272 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538192.153816:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538192.153818:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538192.153819:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538192.153826:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538192.153828:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538192.153828:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538192.153829:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538192.153830:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538192.153831:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538192.153832:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538192.153834:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19991 00000100:00080000:18.0:1584538192.153836:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538192.153845:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538192.153846:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538192.153847:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538192.153848:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538192.153849:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538192.153850:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538192.153850:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538192.153852:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538192.187841:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584538192.187845:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584538192.187847:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584538192.187848:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584538192.187849:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584538192.187851:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584538192.187852:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584538192.187854:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538272 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538192.187883:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078437664 new x1661492078437712 00000800:00020000:2.0:1584538197.916054:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538197.916057:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538197.916059:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538197.916494:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538197.928546:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538192/real 1584538192] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538197.928554:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538197.928555:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538197.928556:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538197.928570:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538197.928572:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538197.928573:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:18.0:1584538197.928574:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:18.0:1584538197.928575:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538197.928577:0:4066:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19956 00000100:00080000:18.0:1584538197.928578:0:4066:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 19979 00000100:00080000:18.0:1584538197.928579:0:4066:0:(import.c:570:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: tried all connections, increasing latency to 6s 00000100:00080000:18.0:1584538197.928581:0:4066:0:(import.c:596:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0001 (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584538197.928582:0:4066:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538197.928587:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538197.928589:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538197.928589:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538197.928590:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538197.928591:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538197.928592:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 19998 00000100:00080000:18.0:1584538197.928594:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538197.928598:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538197.928599:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538197.928599:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538197.928601:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538197.928601:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538197.928602:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538197.928603:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538197.928604:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538197.963121:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584538197.963124:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584538197.963126:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584538197.963127:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584538197.963128:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584538197.963130:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584538197.963131:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584538197.963133:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584538197.963140:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584538197.963142:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584538197.963143:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0001_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584538197.963143:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584538197.963145:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584538197.963146:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584538197.963147:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584538197.963149:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727a80): req@ffff8d2babe46c00 x1661492078436656/t0(0) o17->public1-OST0001-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 456/432 e 0 to 1 dl 1584538150 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 00000100:00080000:18.0:1584538197.963161:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078437712 new x1661492078437776 00000800:00020000:4.0:1584538203.916177:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584538203.916180:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584538203.916182:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538203.916627:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538203.928681:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538197/real 1584538197] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538203.928690:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538203.928691:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538203.928693:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538203.928700:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538203.928701:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538203.928702:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538203.928703:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538203.928704:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538203.928705:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538203.928707:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538203.928709:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20003 00000100:00080000:18.0:1584538203.928711:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538203.928720:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538203.928722:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538203.928722:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538203.928724:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538203.928724:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538203.928725:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538203.928726:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538203.928727:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538203.966865:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584538203.966870:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584538203.966872:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584538203.966873:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584538203.966874:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584538203.966876:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584538203.966878:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584538203.966880:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538203.966909:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078437776 new x1661492078437824 00000800:00020000:2.0:1584538209.916054:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538209.916056:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538209.916058:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538209.916465:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538209.928517:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538203/real 1584538203] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538209.928525:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538209.928526:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538209.928527:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538209.928533:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538209.928534:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538209.928535:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538209.928536:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538209.928538:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538209.928539:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538209.928540:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538209.928542:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20009 00000100:00080000:18.0:1584538209.928544:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538209.928552:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538209.928553:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538209.928554:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538209.928555:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538209.928556:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538209.928557:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538209.928558:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538209.928559:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538209.973356:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584538209.973360:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584538209.973362:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584538209.973362:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584538209.973364:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584538209.973365:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584538209.973367:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584538209.973369:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538209.973399:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078437824 new x1661492078437856 00000800:00000100:0.0:1584538216.141475:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538216.141882:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538216.153942:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538209/real 1584538209] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538216.153952:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538216.153953:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538216.153954:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538216.153961:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538216.153963:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538216.153965:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538216.153966:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538216.153967:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538216.153969:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538216.153969:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538216.153971:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20015 00000100:00080000:18.0:1584538216.153974:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538216.153987:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538216.153989:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538216.153989:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538216.153991:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538216.153991:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538216.153992:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538216.153993:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538216.153995:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538216.201093:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584538216.201099:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584538216.201101:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584538216.201102:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584538216.201104:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584538216.201107:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584538216.201108:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584538216.201110:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538216.201139:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078437856 new x1661492078437888 00000002:00080000:3.0F:1584538216.756242:0:169615:0:(mdc_request.c:903:mdc_close()) @@@ matched open req@ffff8d3bbf9b3180 x1661492078437904/t87765087728(87765087728) o101->public1-MDT0000-mdc-ffff8d3ae8ab1800@10.10.2.11@o2ib:12/10 lens 576/616 e 0 to 0 dl 1584538223 ref 1 fl Complete:RP/4/ffffffff rc 0/-1 00000800:00020000:2.0:1584538221.916072:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538221.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538221.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538221.916590:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538221.928646:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538216/real 1584538216] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538221.928656:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538221.928657:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538221.928659:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538221.928680:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538221.928682:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538221.928687:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538221.928689:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538221.928689:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538221.928691:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538221.928692:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538221.928694:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20022 00000100:00080000:18.0:1584538221.928697:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538221.928704:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538221.928706:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538221.928707:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538221.928708:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538221.928709:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538221.928710:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538221.928711:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538221.928712:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584538221.972189:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584538221.972195:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584538221.972197:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584538221.972199:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584538221.972200:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584538221.972203:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584538221.972204:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584538221.972206:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538221.972236:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078437888 new x1661492078438016 00000800:00020000:0.0:1584538227.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584538227.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584538227.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538227.916505:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538227.928560:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538221/real 1584538221] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538227.928570:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538227.928573:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538227.928575:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538227.928583:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538227.928585:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538227.928586:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538227.928588:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538227.928588:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538227.928590:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538227.928591:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538227.928593:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20027 00000100:00080000:18.0:1584538227.928596:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538227.928612:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538227.928613:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538227.928614:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538227.928616:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538227.928616:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538227.928617:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538227.928618:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538227.928619:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584538227.953568:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584538227.953575:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584538227.953577:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584538227.953579:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584538227.953580:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584538227.953583:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584538227.953585:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584538227.953586:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538227.953616:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078438016 new x1661492078438048 00000800:00020000:2.0:1584538233.916060:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538233.916063:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538233.916065:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538233.916542:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538233.928600:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538227/real 1584538227] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538233.928611:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538233.928612:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538233.928614:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538233.928621:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538233.928624:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538233.928624:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538233.928626:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538233.928626:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538233.928628:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538233.928629:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538233.928631:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20033 00000100:00080000:18.0:1584538233.928634:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538233.928648:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538233.928649:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538233.928650:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538233.928651:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538233.928652:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538233.928653:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538233.928654:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538233.928655:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584538233.973834:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584538233.973841:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584538233.973843:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584538233.973844:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584538233.973847:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584538233.973849:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584538233.973851:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584538233.973853:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538233.973882:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078438048 new x1661492078438096 00000800:00000100:0.0:1584538240.141010:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538240.141441:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538240.153490:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538233/real 1584538233] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538320 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538240.153498:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538240.153499:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538240.153500:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538240.153506:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538240.153507:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538240.153508:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538240.153509:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538240.153510:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538240.153511:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538240.153512:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538240.153514:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20039 00000100:00080000:18.0:1584538240.153516:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538240.153524:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538240.153525:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538240.153526:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538240.153527:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538240.153528:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538240.153529:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538240.153531:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538240.153532:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584538240.206066:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584538240.206069:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584538240.206071:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584538240.206072:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584538240.206073:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584538240.206075:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584538240.206076:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584538240.206078:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538320 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538240.206107:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078438096 new x1661492078438128 00000800:00020000:2.0:1584538245.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538245.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538245.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538245.916491:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538245.928542:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538240/real 1584538240] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538327 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538245.928550:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538245.928551:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538245.928552:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538245.928566:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538245.928567:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538245.928573:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538245.928574:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538245.928575:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538245.928576:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538245.928577:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538245.928578:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20046 00000100:00080000:18.0:1584538245.928580:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538245.928586:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538245.928587:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538245.928588:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538245.928589:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538245.928590:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538245.928591:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538245.928592:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538245.928593:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584538245.985388:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584538245.985392:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584538245.985394:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584538245.985394:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584538245.985396:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584538245.985397:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584538245.985399:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584538245.985401:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538327 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538245.985429:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078438128 new x1661492078438192 00000800:00020000:2.0:1584538251.916067:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538251.916069:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538251.916071:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538251.916543:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538251.928597:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538245/real 1584538245] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538332 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538251.928605:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538251.928606:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538251.928608:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538251.928613:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538251.928615:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538251.928616:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538251.928617:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538251.928617:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538251.928619:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538251.928619:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538251.928621:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20051 00000100:00080000:18.0:1584538251.928623:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538251.928631:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538251.928633:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538251.928634:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538251.928635:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538251.928636:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538251.928637:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538251.928638:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538251.928639:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584538251.979125:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584538251.979128:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584538251.979130:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584538251.979131:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584538251.979133:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584538251.979135:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584538251.979136:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584538251.979138:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538332 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538251.979167:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078438192 new x1661492078438240 00000800:00020000:4.0:1584538257.916171:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584538257.916174:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584538257.916175:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538257.916585:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538257.928637:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538251/real 1584538251] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538338 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538257.928645:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538257.928646:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538257.928647:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538257.928652:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538257.928654:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538257.928654:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538257.928655:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538257.928656:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538257.928657:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538257.928658:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538257.928660:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20057 00000100:00080000:18.0:1584538257.928662:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538257.928672:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538257.928673:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538257.928674:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538257.928675:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538257.928676:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538257.928677:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538257.928677:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538257.928679:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584538257.979591:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584538257.979595:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584538257.979596:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584538257.979597:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584538257.979598:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584538257.979600:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584538257.979602:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584538257.979603:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538338 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538257.979633:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078438240 new x1661492078438272 00000800:00000100:0.0:1584538264.141316:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538264.141750:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538264.153806:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538257/real 1584538257] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538344 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538264.153815:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538264.153816:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538264.153817:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538264.153824:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538264.153826:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538264.153826:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538264.153827:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538264.153828:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538264.153830:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538264.153830:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538264.153833:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20063 00000100:00080000:18.0:1584538264.153835:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538264.153845:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538264.153846:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538264.153847:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538264.153848:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538264.153849:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538264.153850:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538264.153850:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538264.153852:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584538264.204304:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584538264.204308:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584538264.204310:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584538264.204311:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584538264.204312:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584538264.204314:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584538264.204316:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584538264.204318:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538344 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538264.204347:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078438272 new x1661492078438320 00000800:00020000:2.0:1584538269.916058:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538269.916061:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538269.916063:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538269.916471:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538269.928534:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538264/real 1584538264] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538351 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538269.928543:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538269.928544:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538269.928545:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538269.928562:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538269.928564:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538269.928569:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538269.928570:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538269.928571:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538269.928574:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538269.928574:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538269.928576:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20070 00000100:00080000:18.0:1584538269.928578:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538269.928585:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538269.928586:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538269.928587:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538269.928588:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538269.928589:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538269.928590:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538269.928591:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538269.928592:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584538269.973289:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584538269.973294:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584538269.973296:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584538269.973297:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584538269.973299:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584538269.973301:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584538269.973303:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584538269.973305:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538351 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584538269.973313:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078438320 new x1661492078438384 00000800:00020000:2.0:1584538275.916041:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538275.916044:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538275.916046:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538275.916434:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584538275.928484:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538269/real 1584538269] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538356 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584538275.928494:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584538275.928495:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584538275.928497:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584538275.928503:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538275.928505:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584538275.928506:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538275.928507:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584538275.928508:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538275.928510:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584538275.928511:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584538275.928513:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20075 00000100:00080000:4.0:1584538275.928515:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584538275.928531:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538275.928532:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584538275.928533:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538275.928535:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584538275.928536:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538275.928537:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584538275.928538:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538275.928539:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584538275.976503:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584538275.976509:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584538275.976512:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584538275.976513:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584538275.976515:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584538275.976517:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584538275.976519:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584538275.976521:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538356 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584538275.976551:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078438384 new x1661492078438416 00000800:00020000:2.0:1584538281.916106:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538281.916109:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538281.916111:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538281.916504:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584538281.928560:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538275/real 1584538275] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538362 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584538281.928570:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584538281.928572:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584538281.928574:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584538281.928580:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538281.928582:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538281.928583:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538281.928584:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538281.928585:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538281.928587:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584538281.928587:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584538281.928591:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20081 00000100:00080000:6.0:1584538281.928593:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584538281.928607:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538281.928609:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538281.928610:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538281.928611:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538281.928612:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538281.928613:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538281.928614:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538281.928615:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000800:00000400:0.0:1584538283.916074:0:25711:0:(o2iblnd_cb.c:3381:kiblnd_check_conns()) Timed out tx for 10.10.2.22@o2ib: 2 seconds 00000400:00000100:0.0:1584538283.927247:0:25711:0:(lib-msg.c:705:lnet_health_check()) msg 0@<0:0>->10.10.2.22@o2ib exceeded retry count 0 00000100:00080000:12.0:1584538283.927256:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:12.0:1584538283.927258:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0003_UUID on 10.10.2.22@o2ib failed (-110) 00000800:00000100:0.0:1584538286.937064:0:167929:0:(o2iblnd_cb.c:3200:kiblnd_cm_callback()) 10.10.2.22@o2ib: ROUTE ERROR -110 00000100:00080000:6.0:1584538306.969071:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538306.969073:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538306.969078:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538306.969079:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538306.969080:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:6.0:1584538306.969082:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584538306.969083:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584538306.969086:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20087 00000100:00080000:6.0:1584538306.969087:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 18724 00000100:00080000:6.0:1584538306.969103:0:42:0:(import.c:596:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0003 (at 10.10.2.21@o2ib) 00000100:00080000:6.0:1584538306.969104:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.21@o2ib/10.10.2.21@o2ib 00000100:00080000:6.0:1584538306.969113:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538306.969115:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538306.969116:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538306.969117:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538306.969118:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538306.969119:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538306.969119:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538306.969121:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538306.969294:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:14.0:1584538306.969298:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0003_UUID on 10.10.2.21@o2ib failed (-19) 00000100:00080000:6.0:1584538332.057211:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538332.057214:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538332.057220:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538332.057221:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538332.057222:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:6.0:1584538332.057224:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584538332.057225:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584538332.057229:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20087 00000100:00080000:6.0:1584538332.057231:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 20112 00000100:00080000:6.0:1584538332.057233:0:42:0:(import.c:570:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: tried all connections, increasing latency to 6s 00000100:00080000:6.0:1584538332.057236:0:42:0:(import.c:596:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0003 (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584538332.057237:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584538332.057245:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538332.057248:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538332.057249:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538332.057250:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538332.057251:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538332.057252:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538332.057253:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538332.057254:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538332.057469:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584538332.057477:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584538332.057479:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584538332.057481:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584538332.057483:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584538332.057485:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584538332.057487:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584538332.057489:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538362 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584538332.057500:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078438416 new x1661492078438608 00000800:00020000:10.0:1584538337.916118:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:10.0:1584538337.916121:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:10.0:1584538337.916124:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538337.916545:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584538337.928646:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538332/real 1584538332] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538419 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584538337.928656:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584538337.928657:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584538337.928659:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584538337.928667:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538337.928669:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538337.928669:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538337.928671:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538337.928671:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538337.928673:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584538337.928674:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584538337.928676:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20137 00000100:00080000:8.0:1584538337.928678:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584538337.928689:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538337.928690:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538337.928691:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538337.928692:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538337.928694:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538337.928695:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538337.928695:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538337.928697:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538337.974192:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584538337.974197:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584538337.974200:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584538337.974201:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584538337.974203:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584538337.974205:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584538337.974207:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584538337.974209:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538419 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584538337.974238:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078438608 new x1661492078438656 00000800:00000100:0.0:1584538343.141731:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538343.142258:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584538343.154341:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538337/real 1584538337] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538424 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584538343.154349:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584538343.154350:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584538343.154351:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584538343.154357:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538343.154359:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538343.154360:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538343.154361:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538343.154361:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538343.154363:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584538343.154363:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584538343.154365:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20143 00000100:00080000:8.0:1584538343.154367:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584538343.154375:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538343.154376:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538343.154378:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538343.154379:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538343.154380:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538343.154381:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538343.154381:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538343.154382:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538343.196100:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584538343.196104:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584538343.196106:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584538343.196107:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584538343.196108:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584538343.196110:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584538343.196111:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584538343.196113:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538424 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584538343.196123:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078438656 new x1661492078438688 00000800:00000100:0.0:1584538349.141788:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538349.142218:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584538349.154297:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538343/real 1584538343] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538430 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584538349.154305:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584538349.154306:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584538349.154307:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584538349.154313:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538349.154315:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538349.154316:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538349.154317:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538349.154317:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538349.154319:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584538349.154319:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584538349.154321:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20149 00000100:00080000:8.0:1584538349.154323:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584538349.154335:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538349.154336:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538349.154337:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538349.154338:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538349.154339:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538349.154340:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538349.154340:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538349.154342:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538349.206250:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584538349.206254:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584538349.206256:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584538349.206256:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584538349.206258:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584538349.206259:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584538349.206261:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584538349.206263:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538430 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584538349.206293:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078438688 new x1661492078438720 00000800:00020000:2.0:1584538355.916071:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538355.916074:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538355.916076:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538355.916463:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584538355.928532:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538349/real 1584538349] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538436 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584538355.972310:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584538355.972312:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584538355.972313:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584538355.972325:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538355.972327:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538355.972332:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538355.972334:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538355.972334:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538355.972336:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584538355.972336:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584538355.972339:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20155 00000100:00080000:8.0:1584538355.972341:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584538355.972346:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538355.972348:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538355.972348:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538355.972350:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538355.972350:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538355.972351:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538355.972352:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538355.972353:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538356.029813:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584538356.029818:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584538356.029820:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584538356.029821:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584538356.029822:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584538356.029824:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584538356.029825:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584538356.029827:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538436 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584538356.029856:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078438720 new x1661492078438800 00000800:00020000:4.0:1584538361.916152:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584538361.916155:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584538361.916157:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538361.916596:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584538361.928677:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538356/real 1584538356] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538443 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584538361.928685:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584538361.928686:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584538361.928687:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584538361.928692:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538361.928694:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538361.928695:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538361.928696:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538361.928697:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538361.928699:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584538361.928700:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584538361.928701:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20161 00000100:00080000:8.0:1584538361.928703:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584538361.928711:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538361.928713:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538361.928713:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538361.928715:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538361.928715:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538361.928716:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538361.928717:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538361.928718:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538361.977091:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584538361.977095:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584538361.977096:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584538361.977097:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584538361.977098:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584538361.977100:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584538361.977101:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584538361.977103:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538443 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584538361.977132:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078438800 new x1661492078438832 00000800:00000100:0.0:1584538367.142294:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538367.142699:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584538367.154765:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538361/real 1584538361] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538448 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584538367.154772:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584538367.154773:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584538367.154774:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584538367.154779:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538367.154780:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538367.154781:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538367.154783:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538367.154783:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538367.154784:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584538367.154785:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584538367.154786:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20167 00000100:00080000:8.0:1584538367.154788:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584538367.154795:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538367.154796:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538367.154797:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538367.154798:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538367.154799:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538367.154800:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538367.154800:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538367.154812:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538367.201379:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584538367.201383:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584538367.201385:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584538367.201386:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584538367.201387:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584538367.201389:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584538367.201390:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584538367.201392:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538448 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584538367.201404:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078438832 new x1661492078438880 00000800:00000100:0.0:1584538373.142266:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538373.142759:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584538373.154827:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538367/real 1584538367] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538454 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584538373.154835:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584538373.154836:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584538373.154837:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584538373.154842:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538373.154845:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538373.154845:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538373.154847:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538373.154847:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538373.154849:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584538373.154849:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584538373.154851:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20173 00000100:00080000:8.0:1584538373.154853:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584538373.154861:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538373.154862:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538373.154863:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538373.154864:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538373.154865:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538373.154866:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538373.154866:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538373.154867:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538373.196421:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584538373.196425:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584538373.196427:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584538373.196428:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584538373.196430:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584538373.196431:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584538373.196433:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584538373.196435:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538454 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584538373.196464:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078438880 new x1661492078438912 00000800:00000100:0.0:1584538379.143575:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538379.144000:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584538379.156083:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538373/real 1584538373] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538460 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584538379.156090:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584538379.156091:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584538379.156093:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584538379.156105:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538379.156106:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538379.156122:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538379.156123:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538379.156124:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538379.156125:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584538379.156125:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584538379.156127:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20179 00000100:00080000:8.0:1584538379.156129:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584538379.156145:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538379.156146:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538379.156146:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538379.156147:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538379.156148:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538379.156149:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538379.156150:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538379.156151:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538379.205508:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584538379.205512:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584538379.205513:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584538379.205514:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584538379.205515:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584538379.205517:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584538379.205518:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584538379.205520:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538460 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584538379.205553:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078438912 new x1661492078438976 00000800:00020000:6.0:1584538385.916156:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584538385.916160:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584538385.916162:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538385.916561:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584538385.928644:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538379/real 1584538379] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538466 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584538385.928659:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584538385.928660:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584538385.928662:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584538385.928670:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538385.928672:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538385.928673:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538385.928674:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538385.928675:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538385.928676:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584538385.928677:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584538385.928680:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20185 00000100:00080000:8.0:1584538385.928683:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584538385.928696:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538385.928698:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538385.928699:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538385.928700:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538385.928700:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538385.928702:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538385.928702:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538385.928704:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538385.990572:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584538385.990578:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584538385.990580:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584538385.990582:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584538385.990584:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584538385.990586:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584538385.990588:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584538385.990589:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538466 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584538385.990620:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078438976 new x1661492078439024 00000800:00000100:0.0:1584538391.141500:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538391.141964:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584538391.154011:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538385/real 1584538385] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538472 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584538391.154020:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584538391.154021:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584538391.154023:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584538391.154035:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538391.154037:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538391.154038:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538391.154039:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538391.154040:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538391.154041:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584538391.154042:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584538391.154044:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20191 00000100:00080000:8.0:1584538391.154047:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584538391.154060:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538391.154061:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538391.154062:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538391.154063:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538391.154064:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538391.154065:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538391.154078:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538391.154079:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538391.205056:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584538391.205067:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584538391.205072:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584538391.205074:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584538391.205078:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584538391.205083:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584538391.205086:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584538391.205100:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538472 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584538391.205146:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078439024 new x1661492078439056 00000800:00000100:0.0:1584538397.140526:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538397.140999:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584538397.153073:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538391/real 1584538391] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538478 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584538397.153082:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584538397.153084:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584538397.153085:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584538397.153092:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538397.153094:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538397.153095:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538397.153096:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538397.153097:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538397.153098:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584538397.153099:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584538397.153101:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20197 00000100:00080000:8.0:1584538397.153104:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584538397.153115:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538397.153116:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538397.153117:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538397.153118:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538397.153120:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538397.153132:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538397.153133:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538397.153134:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538397.204920:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584538397.204931:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584538397.204934:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584538397.204937:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584538397.204940:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584538397.204945:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584538397.204949:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584538397.204953:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538478 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584538397.205007:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078439056 new x1661492078439104 00000800:00000100:0.0:1584538403.140376:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538403.140836:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584538403.152908:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538397/real 1584538397] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538484 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584538403.152915:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584538403.152917:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584538403.152918:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584538403.152932:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538403.152933:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538403.152950:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538403.152951:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538403.152951:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538403.152953:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584538403.152953:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584538403.152955:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20203 00000100:00080000:8.0:1584538403.152957:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584538403.152973:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538403.152974:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538403.152976:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538403.152977:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538403.152978:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538403.152979:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538403.152979:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538403.152981:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538403.205704:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584538403.205712:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584538403.205715:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584538403.205717:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584538403.205720:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584538403.205725:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584538403.205728:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584538403.205732:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538484 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584538403.205780:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078439104 new x1661492078439168 00000800:00020000:6.0:1584538409.916187:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584538409.916190:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584538409.916192:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538409.916636:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584538409.928713:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538403/real 1584538403] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538490 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584538409.928721:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584538409.928723:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584538409.928724:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584538409.928730:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538409.928732:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538409.928733:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538409.928734:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538409.928735:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538409.928736:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584538409.928737:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584538409.928738:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20209 00000100:00080000:8.0:1584538409.928741:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584538409.928751:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538409.928752:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538409.928753:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538409.928754:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538409.928755:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538409.928756:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538409.928756:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538409.928758:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538409.972528:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584538409.972532:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584538409.972535:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584538409.972536:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584538409.972537:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584538409.972539:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584538409.972541:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584538409.972542:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538490 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584538409.972574:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078439168 new x1661492078439200 00000800:00000100:0.0:1584538415.140540:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538415.141019:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584538415.153114:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538409/real 1584538409] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538496 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584538415.153122:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584538415.153123:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584538415.153124:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584538415.153130:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538415.153131:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538415.153132:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538415.153133:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538415.153134:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538415.153135:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584538415.153136:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584538415.153138:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20215 00000100:00080000:8.0:1584538415.153140:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584538415.153148:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538415.153150:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538415.153150:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538415.153151:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538415.153152:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538415.153153:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538415.153154:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538415.153155:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538415.209264:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584538415.209268:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584538415.209270:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584538415.209271:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584538415.209272:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584538415.209274:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584538415.209276:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584538415.209277:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538496 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584538415.209310:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078439200 new x1661492078439248 00000800:00000100:0.0:1584538421.140685:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538421.141109:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584538421.153164:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538415/real 1584538415] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538502 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584538421.153172:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584538421.153173:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584538421.153174:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584538421.153180:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538421.153181:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538421.153182:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538421.153183:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538421.153184:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538421.153186:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584538421.153187:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584538421.153188:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20221 00000100:00080000:8.0:1584538421.153190:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584538421.153198:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538421.153200:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538421.153200:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538421.153201:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538421.153202:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538421.153203:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538421.153204:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538421.153205:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538421.201275:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584538421.201278:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584538421.201280:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584538421.201281:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584538421.201282:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584538421.201284:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584538421.201285:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584538421.201287:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538502 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584538421.201317:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078439248 new x1661492078439280 00000800:00000100:0.0:1584538427.140736:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538427.141174:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584538427.153238:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538421/real 1584538421] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538508 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584538427.153245:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584538427.178589:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584538427.178591:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584538427.178602:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538427.178603:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538427.178619:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538427.178621:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538427.178621:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538427.178623:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584538427.178623:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584538427.178625:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20227 00000100:00080000:8.0:1584538427.178627:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584538427.178643:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538427.178644:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538427.178645:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538427.178646:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538427.178646:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538427.178647:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538427.178648:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584538427.178649:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538427.216090:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584538427.216095:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584538427.216097:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584538427.216098:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584538427.216099:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584538427.216101:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584538427.216103:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584538427.233882:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538508 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584538427.233926:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078439280 new x1661492078439360 00000800:00020000:4.0:1584538433.916177:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584538433.916179:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584538433.916182:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538433.916624:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584538433.928676:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538427/real 1584538427] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538514 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584538433.928684:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584538433.928685:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584538433.928686:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584538433.928691:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538433.928693:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538433.928694:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538433.928695:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538433.928696:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538433.928697:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584538433.928698:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584538433.928700:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20233 00000100:00080000:10.0:1584538433.928701:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584538433.928711:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538433.928714:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538433.928714:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538433.928715:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538433.928716:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538433.928717:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538433.928718:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538433.928719:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538433.969622:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584538433.969626:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584538433.969627:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584538433.969628:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584538433.969629:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584538433.969631:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584538433.969633:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584538433.969634:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538514 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584538433.969664:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078439360 new x1661492078439392 00000800:00000100:0.0:1584538439.140933:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538439.141366:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584538439.153419:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538433/real 1584538433] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538520 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584538439.153427:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584538439.153428:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584538439.153429:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584538439.153434:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538439.153435:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538439.153436:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538439.153437:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538439.153438:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538439.153439:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584538439.153440:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584538439.153441:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20239 00000100:00080000:10.0:1584538439.153444:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584538439.153453:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538439.153454:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538439.153455:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538439.153456:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538439.153456:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538439.153458:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538439.153458:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538439.153459:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538439.210865:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584538439.210868:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584538439.210870:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584538439.210871:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584538439.210872:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584538439.210874:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584538439.210875:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584538439.210877:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538520 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584538439.210909:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078439392 new x1661492078439424 00000800:00000100:0.0:1584538445.140956:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538445.141465:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584538445.153541:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538439/real 1584538439] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538526 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584538445.153550:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584538445.153551:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584538445.153553:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584538445.153559:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538445.153561:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538445.153562:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538445.153563:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538445.153564:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538445.153565:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584538445.153567:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584538445.153569:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20245 00000100:00080000:10.0:1584538445.153571:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584538445.153581:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538445.153582:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538445.153583:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538445.153584:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538445.153585:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538445.153586:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538445.153586:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538445.153587:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538445.206538:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584538445.206543:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584538445.206546:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584538445.206547:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584538445.206548:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584538445.206550:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584538445.206552:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584538445.206553:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538526 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584538445.206582:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078439424 new x1661492078439472 00000800:00000100:0.0:1584538451.141060:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538451.141528:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584538451.153586:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538445/real 1584538445] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538532 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584538451.153597:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584538451.153599:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584538451.153601:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584538451.153624:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538451.153626:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538451.153632:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538451.153633:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538451.153635:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538451.153636:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584538451.153637:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584538451.153640:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20251 00000100:00080000:10.0:1584538451.153643:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584538451.153651:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538451.153652:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538451.153653:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538451.153655:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538451.153655:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538451.153657:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538451.153657:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584538451.153659:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538451.200811:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538451.200822:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538451.200826:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538451.200829:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538451.200833:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538451.200837:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538451.200841:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538451.200845:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538532 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538451.200893:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078439472 new x1661492078439536 00000800:00020000:4.0:1584538457.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584538457.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584538457.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538457.916558:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538457.928614:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538451/real 1584538451] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538538 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538457.928624:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538457.928626:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538457.928628:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538457.928635:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538457.928637:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538457.928638:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538457.928640:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538457.928640:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538457.928642:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538457.928643:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538457.928645:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20257 00000100:00080000:14.0:1584538457.928648:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538457.928660:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538457.928663:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538457.928664:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538457.928665:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538457.928666:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538457.928667:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538457.928668:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538457.928669:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538457.978997:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538457.979003:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538457.979006:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538457.979007:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538457.979009:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538457.979012:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538457.979013:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538457.979015:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538538 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538457.979045:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078439536 new x1661492078439584 00000800:00000100:0.0:1584538463.140967:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538463.141563:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538463.153613:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538457/real 1584538457] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538544 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538463.153621:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538463.153622:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538463.153623:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538463.153628:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538463.153630:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538463.153631:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538463.153632:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538463.153633:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538463.153634:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538463.153635:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538463.153636:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20263 00000100:00080000:14.0:1584538463.153639:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538463.153648:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538463.153649:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538463.153650:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538463.153651:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538463.153652:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538463.153653:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538463.153653:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538463.153654:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538463.161540:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538463.161544:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538463.161545:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538463.161546:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538463.161548:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538463.161550:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538463.161552:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538463.161554:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538544 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538463.161582:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078439584 new x1661492078439616 00000800:00000100:0.0:1584538469.141045:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538469.141509:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538469.153561:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538463/real 1584538463] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538550 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538469.153569:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538469.153570:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538469.153572:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538469.153578:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538469.153579:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538469.153580:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538469.153581:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538469.153582:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538469.153583:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538469.153585:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538469.153587:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20269 00000100:00080000:14.0:1584538469.153589:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538469.153598:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538469.153599:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538469.153600:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538469.153601:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538469.153602:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538469.153603:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538469.153604:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538469.153605:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538469.161366:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538469.161370:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538469.161372:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538469.161373:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538469.161374:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538469.161376:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538469.161378:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538469.161379:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538550 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538469.161408:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078439616 new x1661492078439648 00000800:00000100:0.0:1584538475.141160:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538475.141595:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538475.153651:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538469/real 1584538469] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538556 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538475.153659:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538475.153660:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538475.153661:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538475.153676:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538475.153677:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538475.153683:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538475.153684:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538475.153685:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538475.153687:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538475.153687:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538475.153689:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20275 00000100:00080000:14.0:1584538475.153691:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538475.153697:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538475.153698:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538475.153699:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538475.153700:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538475.153701:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538475.153702:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538475.153703:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538475.153704:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538475.162024:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538475.162049:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538475.162051:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538475.162052:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538475.162053:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538475.162055:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538475.162057:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538475.162058:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538556 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538475.162087:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078439648 new x1661492078439728 00000800:00020000:2.0:1584538481.916063:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538481.916066:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538481.916068:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538481.916480:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538481.928533:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538475/real 1584538475] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538562 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538481.928540:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538481.928541:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538481.928543:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538481.928548:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538481.928549:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538481.928551:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538481.928552:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538481.928552:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538481.928554:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538481.928554:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538481.928556:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20281 00000100:00080000:14.0:1584538481.928558:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538481.928566:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538481.928567:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538481.928568:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538481.928569:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538481.928570:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538481.928571:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538481.928571:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538481.928572:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538481.936779:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538481.936783:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538481.936784:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538481.936785:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538481.936787:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538481.936789:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538481.936790:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538481.936792:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538562 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538481.936821:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078439728 new x1661492078439760 00000800:00000100:0.0:1584538487.141335:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538487.141797:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538487.153869:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538481/real 1584538481] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538568 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538487.153876:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538487.153877:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538487.153878:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538487.153884:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538487.153886:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538487.153886:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538487.153887:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538487.153888:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538487.153889:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538487.153890:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538487.153891:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20287 00000100:00080000:14.0:1584538487.153893:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538487.153901:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538487.153902:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538487.153902:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538487.153903:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538487.153904:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538487.153905:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538487.153906:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538487.153918:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538487.162155:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538487.162159:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538487.162161:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538487.162162:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538487.162163:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538487.162165:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538487.162166:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538487.162168:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538568 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538487.162215:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078439760 new x1661492078439808 00000800:00000100:0.0:1584538493.141400:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538493.141962:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538493.154016:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538487/real 1584538487] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538574 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538493.154056:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538493.154058:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538493.154060:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538493.154068:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538493.154070:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538493.154070:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538493.154072:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538493.154072:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538493.154075:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538493.154075:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538493.154077:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20293 00000100:00080000:14.0:1584538493.154080:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538493.154095:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538493.154097:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538493.154097:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538493.154099:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538493.154099:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538493.154101:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538493.154102:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538493.154103:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538493.162185:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538493.162191:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538493.162194:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538493.162195:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538493.162197:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538493.162200:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538493.162201:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538493.162203:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538574 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538493.162213:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078439808 new x1661492078439840 00000800:00000100:0.0:1584538499.141548:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538499.141998:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538499.154053:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538493/real 1584538493] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538580 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538499.154062:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538499.154063:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538499.154065:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538499.154081:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538499.154083:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538499.154089:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538499.154090:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538499.154091:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538499.154092:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538499.154093:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538499.154095:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20299 00000100:00080000:14.0:1584538499.154097:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538499.154103:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538499.154104:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538499.154105:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538499.154106:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538499.154106:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538499.154108:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538499.154108:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538499.154110:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538499.161884:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538499.161889:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538499.161891:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538499.161891:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538499.161894:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538499.161896:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538499.161898:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538499.161900:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538580 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538499.161929:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078439840 new x1661492078439904 00000800:00020000:4.0:1584538505.916082:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584538505.916086:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584538505.916088:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538505.916504:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538505.928562:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538499/real 1584538499] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538586 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538505.928573:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538505.928574:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538505.928576:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538505.928582:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538505.928584:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538505.928585:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538505.928586:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538505.928587:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538505.928589:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538505.928589:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538505.928591:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20305 00000100:00080000:14.0:1584538505.928595:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538505.928608:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538505.928610:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538505.928611:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538505.928612:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538505.928613:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538505.928614:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538505.928616:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538505.928617:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538505.936360:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538505.936367:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538505.936369:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538505.936371:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538505.936373:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538505.936375:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538505.936377:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538505.936379:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538586 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538505.936388:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078439904 new x1661492078439952 00000800:00000100:0.0:1584538511.140723:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538511.141181:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538511.153233:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538505/real 1584538505] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538592 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538511.153244:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538511.153246:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538511.153247:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538511.153254:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538511.153256:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538511.153257:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538511.153258:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538511.153259:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538511.153261:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538511.153262:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538511.153264:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20311 00000100:00080000:14.0:1584538511.153267:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538511.153283:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538511.153285:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538511.153286:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538511.153287:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538511.153289:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538511.153290:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538511.153291:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538511.153292:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538511.161394:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538511.161401:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538511.161403:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538511.161405:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538511.161407:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538511.161409:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538511.161411:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538511.161413:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538592 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538511.161442:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078439952 new x1661492078439984 00000800:00000100:0.0:1584538517.140578:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538517.141081:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538517.153139:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538511/real 1584538511] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538598 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538517.153147:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538517.153148:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538517.153150:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538517.153156:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538517.153158:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538517.153159:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538517.153160:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538517.153161:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538517.153162:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538517.153163:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538517.153164:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20317 00000100:00080000:14.0:1584538517.153167:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538517.153177:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538517.153178:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538517.153180:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538517.153181:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538517.153182:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538517.153183:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538517.153184:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538517.153185:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538517.161116:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538517.161121:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538517.161123:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538517.161124:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538517.161127:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538517.161129:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538517.161131:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538517.161132:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538598 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538517.161164:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078439984 new x1661492078440048 00000800:00000100:0.0:1584538523.140790:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538523.141230:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538523.153280:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538517/real 1584538517] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538604 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538523.153288:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538523.153289:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538523.153290:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538523.153304:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538523.153306:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538523.153315:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538523.153316:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538523.153317:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538523.153318:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538523.153319:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538523.153320:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20323 00000100:00080000:14.0:1584538523.153322:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538523.153328:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538523.153329:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538523.153330:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538523.153331:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538523.153332:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538523.153333:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538523.153333:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538523.153335:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538523.161471:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538523.161475:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538523.161477:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538523.161478:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538523.161479:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538523.161481:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538523.161483:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538523.161484:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538604 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538523.161513:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078440048 new x1661492078440112 00000800:00000100:0.0:1584538529.140761:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538529.141178:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538529.153230:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538523/real 1584538523] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538610 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538529.153238:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538529.153239:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538529.153241:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538529.153245:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538529.153247:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538529.153248:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538529.153249:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538529.153249:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538529.153251:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538529.153251:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538529.153254:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20329 00000100:00080000:14.0:1584538529.153256:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538529.153264:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538529.153265:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538529.153266:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538529.153267:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538529.153268:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538529.153269:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538529.153269:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538529.153270:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538529.161202:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538529.161206:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538529.161207:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538529.161208:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538529.161209:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538529.161211:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538529.161213:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538529.161214:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538610 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538529.161244:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078440112 new x1661492078440144 00000800:00000100:0.0:1584538535.140898:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538535.141319:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538535.153370:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538529/real 1584538529] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538616 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538535.153378:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538535.153379:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538535.153381:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538535.153386:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538535.153387:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538535.153388:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538535.153389:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538535.153390:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538535.153392:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538535.153393:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538535.153394:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20335 00000100:00080000:14.0:1584538535.153396:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538535.153405:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538535.153406:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538535.153407:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538535.153408:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538535.153408:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538535.153409:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538535.153410:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538535.153411:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538535.161514:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538535.161519:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538535.161520:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538535.161521:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538535.161523:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538535.161525:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538535.161527:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538535.161528:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538616 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538535.161558:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078440144 new x1661492078440192 00000800:00000100:0.0:1584538541.140933:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538541.141342:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538541.153393:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538535/real 1584538535] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538622 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538541.153401:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538541.153402:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538541.153404:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538541.153408:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538541.153410:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538541.153410:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538541.153412:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538541.153413:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538541.153414:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538541.153415:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538541.153416:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20341 00000100:00080000:14.0:1584538541.153418:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538541.153427:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538541.153428:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538541.153429:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538541.153430:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538541.153430:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538541.153431:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538541.153432:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538541.153433:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538541.161232:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538541.161236:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538541.161237:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538541.161238:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538541.161239:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538541.161241:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538541.161243:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538541.161245:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538622 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538541.161253:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078440192 new x1661492078440224 00000800:00000100:0.0:1584538547.141011:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538547.141584:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538547.153672:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538541/real 1584538541] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538628 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538547.153679:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538547.153680:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538547.153681:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538547.153696:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538547.153698:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538547.153701:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538547.153702:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538547.153714:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538547.153716:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538547.153716:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538547.153718:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20347 00000100:00080000:14.0:1584538547.153720:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538547.153725:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538547.153727:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538547.153727:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538547.153728:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538547.153729:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538547.153730:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538547.153731:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538547.153732:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538547.161842:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538547.161847:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538547.161848:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538547.161849:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538547.161851:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538547.161852:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538547.161854:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538547.161856:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538628 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538547.161884:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078440224 new x1661492078440304 00000800:00000100:0.0:1584538553.141117:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538553.141554:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538553.153610:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538547/real 1584538547] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538634 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538553.153617:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538553.153618:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538553.153621:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538553.153626:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538553.153628:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538553.153629:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538553.153630:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538553.153630:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538553.153632:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538553.153632:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538553.153634:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20353 00000100:00080000:14.0:1584538553.153636:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538553.153644:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538553.153645:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538553.153646:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538553.153647:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538553.153648:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538553.153649:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538553.153650:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538553.153651:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538553.161640:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538553.161644:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538553.161646:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538553.161646:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538553.161649:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538553.161651:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538553.161652:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538553.161654:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538634 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538553.161662:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078440304 new x1661492078440336 00000800:00000100:0.0:1584538559.141211:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538559.141631:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538559.153713:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538553/real 1584538553] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538640 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538559.153725:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538559.153726:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538559.153727:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538559.153733:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538559.153735:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538559.153736:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538559.153737:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538559.153737:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538559.153739:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538559.153739:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538559.153741:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20359 00000100:00080000:14.0:1584538559.153743:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538559.153751:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538559.153752:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538559.153753:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538559.153754:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538559.153755:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538559.153756:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538559.153756:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538559.153758:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538559.161652:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538559.161656:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538559.161657:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538559.161659:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538559.161660:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538559.161662:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538559.161663:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538559.161665:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538640 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538559.161694:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078440336 new x1661492078440368 00000800:00000100:0.0:1584538565.141302:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538565.141688:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538565.153744:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538559/real 1584538559] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538646 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538565.153754:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538565.153755:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538565.153757:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538565.153763:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538565.153765:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538565.153766:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538565.153768:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538565.153768:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538565.153770:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538565.153771:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538565.153773:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20365 00000100:00080000:14.0:1584538565.153776:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538565.153787:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538565.153789:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538565.153790:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538565.153791:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538565.153792:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538565.153793:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538565.153795:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538565.153796:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538565.161699:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538565.161705:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538565.161707:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538565.161709:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538565.161711:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538565.161713:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538565.161714:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538565.161716:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538646 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584538565.161745:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078440368 new x1661492078440416 00000800:00000100:0.0:1584538571.141396:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538571.141822:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584538571.153876:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538565/real 1584538565] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538652 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584538571.153886:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584538571.153888:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584538571.153889:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584538571.153909:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538571.153911:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538571.153919:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538571.153920:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538571.153921:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538571.153922:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584538571.153923:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584538571.153925:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20371 00000100:00080000:14.0:1584538571.153928:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584538571.153935:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538571.153937:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538571.153937:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538571.153939:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538571.153940:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538571.153941:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538571.153942:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584538571.153943:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538571.161987:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538571.161993:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538571.161995:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538571.161997:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538571.161999:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538571.162002:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538571.162003:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538571.162005:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538652 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538571.162036:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078440416 new x1661492078440480 00000800:00000100:0.0:1584538577.140741:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538577.141195:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538577.153283:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538571/real 1584538571] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538658 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538577.153292:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538577.153294:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538577.153296:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538577.153303:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538577.153305:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538577.153305:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538577.153308:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538577.153308:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538577.153310:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538577.153311:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538577.153313:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20377 00000100:00080000:18.0:1584538577.153315:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538577.153327:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538577.153328:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538577.153329:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538577.153330:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538577.153331:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538577.153332:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538577.153333:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538577.153334:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538577.161579:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538577.161585:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538577.161587:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538577.161589:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538577.161590:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538577.161593:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538577.161595:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538577.161597:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538658 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538577.161625:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078440480 new x1661492078440528 00000800:00000100:0.0:1584538583.141098:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538583.141568:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538583.153642:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538577/real 1584538577] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538664 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538583.153651:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538583.153653:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538583.153654:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538583.153661:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538583.153665:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538583.153666:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538583.153667:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538583.153668:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538583.153670:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538583.153671:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538583.153673:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20383 00000100:00080000:18.0:1584538583.153675:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538583.153689:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538583.153690:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538583.153691:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538583.153692:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538583.153692:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538583.153694:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538583.153694:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538583.153696:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538583.162010:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538583.162016:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538583.162018:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538583.162019:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538583.162021:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538583.162024:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538583.162025:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538583.162034:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538664 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538583.162064:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078440528 new x1661492078440560 00000800:00000100:0.0:1584538589.140936:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538589.141397:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538589.153468:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538583/real 1584538583] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538670 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538589.153476:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538589.153478:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538589.153480:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538589.153487:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538589.153489:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538589.153490:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538589.153492:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538589.153492:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538589.153494:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538589.153495:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538589.153496:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20389 00000100:00080000:18.0:1584538589.153499:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538589.153511:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538589.153512:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538589.153513:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538589.153514:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538589.153515:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538589.153516:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538589.153517:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538589.153518:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538589.161686:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538589.161692:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538589.161694:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538589.161695:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538589.161697:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538589.161700:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538589.161701:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538589.161703:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538670 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538589.161733:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078440560 new x1661492078440592 00000800:00000100:0.0:1584538595.141297:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538595.141809:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538595.153895:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538589/real 1584538589] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538676 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538595.153907:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538595.153908:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538595.153909:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538595.153925:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538595.153927:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538595.153930:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538595.153931:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538595.153932:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538595.153945:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538595.153946:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538595.153947:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20395 00000100:00080000:18.0:1584538595.153949:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538595.153956:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538595.153957:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538595.153958:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538595.153959:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538595.153960:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538595.153961:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538595.153962:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538595.153963:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538595.161950:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538595.161956:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538595.161957:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538595.161958:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538595.161960:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538595.161962:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538595.161964:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538595.161965:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538676 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538595.161994:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078440592 new x1661492078440672 00000800:00000100:0.0:1584538601.141204:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538601.141672:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538601.153745:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538595/real 1584538595] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538682 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538601.153752:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538601.153753:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538601.153755:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538601.153760:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538601.153762:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538601.153763:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538601.153764:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538601.153764:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538601.153766:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538601.153766:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538601.153768:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20401 00000100:00080000:18.0:1584538601.153770:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538601.153778:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538601.153779:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538601.153780:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538601.153781:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538601.153781:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538601.153783:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538601.153784:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538601.153797:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538601.161728:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538601.161732:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538601.161734:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538601.161734:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538601.161736:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538601.161738:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538601.161739:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538601.161741:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538682 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538601.161770:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078440672 new x1661492078440704 00000800:00000100:0.0:1584538607.141162:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538607.141601:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538607.153672:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538601/real 1584538601] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538688 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538607.153679:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538607.153680:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538607.153681:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538607.153686:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538607.153687:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538607.153688:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538607.153689:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538607.153690:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538607.153691:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538607.153692:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538607.153693:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20407 00000100:00080000:18.0:1584538607.153695:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538607.153703:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538607.153704:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538607.153705:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538607.153706:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538607.153707:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538607.153708:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538607.153709:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538607.153721:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538607.161982:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538607.161986:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538607.161988:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538607.161988:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538607.161991:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538607.161993:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538607.161995:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538607.161997:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538688 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538607.162036:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078440704 new x1661492078440752 00000800:00000100:0.0:1584538613.141252:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538613.141679:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538613.153750:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538607/real 1584538607] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538694 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538613.153758:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538613.153759:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538613.153760:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538613.153766:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538613.153768:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538613.153769:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538613.153770:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538613.153770:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538613.153772:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538613.153772:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538613.153774:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20413 00000100:00080000:18.0:1584538613.153776:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538613.153784:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538613.153785:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538613.153787:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538613.153788:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538613.153789:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538613.153790:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538613.153790:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538613.153803:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538613.161800:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538613.161804:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538613.161805:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538613.161806:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538613.161807:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538613.161809:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538613.161811:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538613.161813:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538694 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538613.161841:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078440752 new x1661492078440784 00000800:00000100:0.0:1584538619.141596:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538619.142112:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538619.154184:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538613/real 1584538613] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538700 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538619.154192:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538619.154193:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538619.154194:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538619.154207:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538619.154209:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538619.154211:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538619.154224:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538619.154224:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538619.154226:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538619.154226:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538619.154228:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20419 00000100:00080000:18.0:1584538619.154230:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538619.154247:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538619.154248:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538619.154249:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538619.154250:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538619.154251:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538619.154252:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538619.154253:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538619.154254:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538619.162241:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538619.162245:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538619.162246:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538619.162247:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538619.162248:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538619.162250:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538619.162252:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538619.162253:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538700 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538619.162284:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078440784 new x1661492078440848 00000800:00000100:0.0:1584538625.141454:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538625.141900:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538625.153988:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538619/real 1584538619] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538706 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538625.153999:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538625.154001:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538625.154002:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538625.154011:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538625.154013:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538625.154014:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538625.154015:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538625.154016:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538625.154018:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538625.154019:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538625.154022:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20425 00000100:00080000:18.0:1584538625.154025:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538625.154055:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538625.154056:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538625.154057:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538625.154058:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538625.154059:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538625.154060:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538625.154061:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538625.154062:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538625.162312:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538625.162319:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538625.162321:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538625.162322:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538625.162325:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538625.162328:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538625.162329:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538625.162331:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538706 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538625.162363:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078440848 new x1661492078440896 00000800:00020000:4.0:1584538631.916096:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584538631.938908:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584538631.963084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538631.963590:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538631.975733:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538625/real 1584538625] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538712 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538631.975742:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538631.975744:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538631.975745:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538631.975752:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538631.975754:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538631.975755:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538631.975756:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538631.975757:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538631.975760:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538631.975760:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538631.975763:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20431 00000100:00080000:18.0:1584538631.975765:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538631.975778:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538631.975779:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538631.975780:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538631.975781:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538631.975782:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538631.975783:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538631.975784:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538631.975785:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538631.984126:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538631.984132:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538631.984134:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538631.984135:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538631.984137:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538631.984139:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538631.984141:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538631.984143:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538712 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538631.984173:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078440896 new x1661492078440928 00000800:00000100:0.0:1584538637.141721:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538637.142128:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538637.154216:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538631/real 1584538631] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538718 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538637.154225:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538637.154226:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538637.154228:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538637.154235:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538637.154237:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538637.154238:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538637.154240:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538637.154241:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538637.154243:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538637.154243:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538637.154245:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20437 00000100:00080000:18.0:1584538637.154248:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538637.154258:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538637.154260:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538637.154260:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538637.154262:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538637.154262:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538637.154264:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538637.154264:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538637.154277:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538637.198875:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538637.198882:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538637.198884:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538637.198885:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538637.198887:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538637.198889:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538637.198891:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538637.198893:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538718 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538637.198944:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078440928 new x1661492078440976 00000800:00000100:0.0:1584538643.140517:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538643.141054:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538643.153137:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538637/real 1584538637] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538724 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538643.153148:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538643.153149:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538643.153151:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538643.153173:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538643.153178:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538643.153192:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538643.153193:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538643.153194:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538643.153196:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538643.153197:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538643.153210:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20443 00000100:00080000:18.0:1584538643.153213:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538643.153221:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538643.153222:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538643.153223:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538643.153224:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538643.153225:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538643.153226:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538643.153227:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538643.153228:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538643.161264:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538643.161271:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538643.161274:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538643.161275:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538643.161278:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538643.161281:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538643.161282:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538643.161284:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538724 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538643.161314:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078440976 new x1661492078441040 00000800:00000100:0.0:1584538649.140768:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538649.141226:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538649.153293:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538643/real 1584538643] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538730 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538649.153301:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538649.153302:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538649.153305:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538649.153311:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538649.153312:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538649.153313:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538649.153314:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538649.153315:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538649.153317:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538649.153317:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538649.153319:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20449 00000100:00080000:18.0:1584538649.153321:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538649.153331:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538649.153332:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538649.153333:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538649.153334:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538649.153334:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538649.153336:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538649.153336:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538649.153337:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538649.162242:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538649.162246:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538649.162247:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538649.162248:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538649.162249:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538649.162251:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538649.162253:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538649.162255:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538730 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538649.162284:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078441040 new x1661492078441072 00000800:00020000:6.0:1584538655.916188:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584538655.916191:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584538655.916193:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538655.916665:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538655.928718:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538649/real 1584538649] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538736 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538655.928731:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538655.928732:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538655.928733:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538655.928739:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538655.928740:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538655.928741:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538655.928742:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538655.928743:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538655.928745:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538655.928745:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538655.928747:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20455 00000100:00080000:18.0:1584538655.928749:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538655.928758:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538655.928759:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538655.928760:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538655.928761:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538655.928762:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538655.928763:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538655.928764:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538655.928765:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538655.936712:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538655.936717:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538655.936718:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538655.936719:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538655.936720:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538655.936723:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538655.936724:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538655.936726:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538736 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538655.936756:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078441072 new x1661492078441120 00000800:00000100:0.0:1584538661.140899:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538661.141364:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538661.153414:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538655/real 1584538655] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538742 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538661.153421:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538661.153423:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538661.153424:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538661.153429:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538661.153431:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538661.153432:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538661.153433:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538661.153433:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538661.153435:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538661.153436:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538661.153437:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20461 00000100:00080000:18.0:1584538661.153439:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538661.153448:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538661.153449:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538661.153450:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538661.153451:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538661.153451:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538661.153452:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538661.153454:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538661.153455:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538661.161500:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538661.161504:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538661.161506:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538661.161506:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538661.161509:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538661.161510:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538661.161512:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538661.161513:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538742 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538661.161545:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078441120 new x1661492078441152 00000800:00000100:0.0:1584538667.141062:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538667.141527:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538667.153592:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538661/real 1584538661] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538748 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538667.153599:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538667.153600:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538667.153601:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538667.153614:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538667.153616:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538667.153618:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538667.153631:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538667.153631:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538667.153633:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538667.153633:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538667.153635:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20467 00000100:00080000:18.0:1584538667.153637:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538667.153653:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538667.153655:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538667.153655:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538667.153656:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538667.153658:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538667.153659:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538667.153659:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538667.153660:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584538667.161605:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584538667.161613:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584538667.161616:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584538667.161618:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584538667.161621:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584538667.161625:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584538667.161629:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584538667.161633:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538748 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538667.161680:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078441152 new x1661492078441232 00000800:00000100:0.0:1584538673.141047:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538673.141456:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538673.153506:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538667/real 1584538667] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538754 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538673.153513:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538673.153515:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538673.153516:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538673.153522:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538673.153523:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538673.153524:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538673.153525:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538673.153526:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538673.153527:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538673.153528:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538673.153529:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20473 00000100:00080000:18.0:1584538673.153531:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538673.153540:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538673.153541:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538673.153543:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538673.153544:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538673.153545:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538673.153546:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538673.153546:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538673.153547:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584538673.161731:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584538673.161735:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584538673.161736:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584538673.161737:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584538673.161739:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584538673.161741:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584538673.161743:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584538673.161744:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538754 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538673.161774:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078441232 new x1661492078441264 00000800:00020000:4.0:1584538679.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584538679.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584538679.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538679.916532:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538679.928582:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538673/real 1584538673] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538760 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538679.928589:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538679.928590:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538679.928592:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538679.928597:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538679.928598:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538679.928599:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538679.928600:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538679.928601:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538679.928602:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538679.928603:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538679.928604:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20479 00000100:00080000:18.0:1584538679.928606:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538679.928615:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538679.928617:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538679.928617:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538679.928618:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538679.928619:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538679.928620:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538679.928621:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538679.928622:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584538679.937400:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584538679.937404:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584538679.937405:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584538679.937406:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584538679.937407:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584538679.937409:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584538679.937411:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584538679.937412:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538760 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538679.937421:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078441264 new x1661492078441296 00000800:00000100:0.0:1584538685.141245:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538685.141702:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538685.153756:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538679/real 1584538679] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538766 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538685.153765:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538685.153767:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538685.153769:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538685.153775:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538685.153777:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538685.153778:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538685.153779:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538685.153780:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538685.153782:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538685.153783:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538685.153785:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20485 00000100:00080000:18.0:1584538685.153788:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538685.153802:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538685.153803:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538685.153804:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538685.153805:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538685.153806:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538685.153807:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538685.153807:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538685.153809:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584538685.382203:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584538685.382210:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584538685.382212:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584538685.382213:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584538685.382215:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584538685.382218:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584538685.382219:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584538685.382221:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538766 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538685.382251:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078441296 new x1661492078441344 00000800:00000100:0.0:1584538691.141386:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538691.141827:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538691.153904:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538685/real 1584538685] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538772 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538691.153916:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538691.153917:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538691.153919:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538691.153943:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538691.153945:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538691.153952:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538691.153953:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538691.153954:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538691.153957:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538691.153958:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538691.153960:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20491 00000100:00080000:18.0:1584538691.153963:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538691.153971:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538691.153973:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538691.153974:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538691.153975:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538691.153976:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538691.153977:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538691.153978:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538691.153979:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584538691.175204:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584538691.175211:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584538691.175213:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584538691.175215:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584538691.175218:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584538691.175221:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584538691.175222:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584538691.175224:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538772 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538691.175255:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078441344 new x1661492078441408 00000800:00000100:0.0:1584538697.140883:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538697.141347:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538697.153419:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538691/real 1584538691] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538778 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538697.153428:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538697.153430:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538697.153432:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538697.153439:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538697.153441:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538697.153442:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538697.153444:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538697.153445:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538697.153446:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538697.153447:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538697.153449:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20497 00000100:00080000:18.0:1584538697.153452:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538697.153464:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538697.153465:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538697.153466:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538697.153467:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538697.153468:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538697.153469:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538697.153470:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538697.153482:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584538697.195637:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584538697.195644:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584538697.195646:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584538697.195647:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584538697.195649:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584538697.195652:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584538697.195654:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584538697.195656:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538778 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538697.195685:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078441408 new x1661492078441456 00000800:00020000:4.0:1584538703.916072:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584538703.916075:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584538703.916077:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538703.916489:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538703.928545:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538697/real 1584538697] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538784 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538703.928554:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538703.928555:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538703.928557:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538703.928563:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538703.928565:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538703.928566:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538703.928567:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538703.928568:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538703.928570:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538703.928571:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538703.928572:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20503 00000100:00080000:18.0:1584538703.928575:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538703.928587:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538703.928588:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538703.928589:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538703.928590:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538703.928591:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538703.928592:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538703.928593:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538703.928594:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584538703.945969:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584538703.945975:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584538703.945977:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584538703.945979:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584538703.945980:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584538703.945983:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584538703.945984:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584538703.945986:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538784 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538703.945995:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078441456 new x1661492078441488 00000800:00000100:0.0:1584538709.140955:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538709.141372:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538709.153422:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538703/real 1584538703] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538790 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538709.153430:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538709.153431:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538709.153433:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538709.153438:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538709.153439:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538709.153440:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538709.153441:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538709.153442:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538709.153443:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538709.153444:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538709.153446:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20509 00000100:00080000:18.0:1584538709.153448:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538709.153456:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538709.153457:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538709.153458:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538709.153459:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538709.153460:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538709.153461:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538709.153462:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538709.153463:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584538709.174197:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584538709.174201:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584538709.174202:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584538709.174203:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584538709.174205:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584538709.174207:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584538709.174209:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584538709.174211:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538790 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584538709.174240:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078441488 new x1661492078441520 00000800:00000100:0.0:1584538715.141180:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538715.141612:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584538715.153662:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538709/real 1584538709] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538796 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584538715.153675:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584538715.153676:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584538715.153677:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584538715.153692:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538715.153693:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538715.153702:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538715.153704:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538715.153704:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538715.153706:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584538715.153706:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584538715.153708:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20515 00000100:00080000:18.0:1584538715.153711:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584538715.153716:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538715.153717:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538715.153718:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538715.153719:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538715.153720:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538715.153721:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538715.153721:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584538715.153723:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584538715.172590:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584538715.172594:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584538715.172596:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584538715.172597:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584538715.172598:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584538715.172600:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584538715.172602:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584538715.172603:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538796 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584538715.172614:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078441520 new x1661492078441600 00000800:00000100:0.0:1584538721.141055:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538721.141513:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584538721.153563:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538715/real 1584538715] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538802 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584538721.153571:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584538721.153572:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584538721.153574:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584538721.153579:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538721.153581:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538721.153582:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538721.153583:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538721.153585:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538721.153587:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584538721.153587:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584538721.153589:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20521 00000100:00080000:22.0:1584538721.153591:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584538721.153600:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538721.153602:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538721.153602:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538721.153604:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538721.153604:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538721.153605:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538721.153606:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538721.153607:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584538721.180453:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584538721.180456:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584538721.180458:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584538721.180459:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584538721.180460:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584538721.180462:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584538721.180463:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584538721.180465:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538802 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584538721.180495:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078441600 new x1661492078441632 00000800:00020000:6.0:1584538727.916091:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584538727.916094:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584538727.916096:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538727.916562:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584538727.928621:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538721/real 1584538721] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538808 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584538727.928629:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584538727.928630:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584538727.928632:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584538727.928638:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538727.928640:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538727.928642:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538727.928643:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538727.928644:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538727.928645:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584538727.928646:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584538727.928647:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20527 00000100:00080000:22.0:1584538727.928649:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584538727.928658:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538727.928660:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538727.928660:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538727.928661:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538727.928662:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538727.928663:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538727.928664:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538727.928665:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584538727.949851:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584538727.949855:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584538727.949857:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584538727.949858:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584538727.949860:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584538727.949862:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584538727.949864:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584538727.949865:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538808 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584538727.949895:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078441632 new x1661492078441680 00000800:00000100:0.0:1584538733.141215:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538733.141741:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584538733.153790:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538727/real 1584538727] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538814 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584538733.153798:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584538733.153799:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584538733.153800:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584538733.153806:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538733.153808:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538733.153809:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538733.153810:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538733.153811:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538733.153812:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584538733.153813:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584538733.153814:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20533 00000100:00080000:22.0:1584538733.153816:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584538733.153824:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538733.153826:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538733.153826:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538733.153828:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538733.153828:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538733.153829:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538733.153830:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538733.153831:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584538733.170285:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584538733.170289:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584538733.170291:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584538733.170291:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584538733.170293:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584538733.170295:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584538733.170296:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584538733.170298:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538814 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584538733.170326:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078441680 new x1661492078441712 00000800:00000100:0.0:1584538739.141501:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538739.141935:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584538739.153984:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538733/real 1584538733] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538820 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584538739.153992:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584538739.153994:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584538739.153995:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584538739.154010:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538739.154011:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538739.154019:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538739.154021:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538739.154021:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538739.154023:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584538739.154023:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584538739.154025:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20539 00000100:00080000:22.0:1584538739.154044:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584538739.154050:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538739.154052:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538739.154052:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538739.154053:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538739.154054:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538739.154055:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538739.154056:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538739.154057:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584538739.173177:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584538739.173181:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584538739.173183:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584538739.173183:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584538739.173185:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584538739.173187:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584538739.173188:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584538739.173190:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538820 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584538739.173200:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078441712 new x1661492078441776 00000800:00000100:0.0:1584538745.141616:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538745.142100:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584538745.154160:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538739/real 1584538739] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538826 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584538745.154172:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584538745.154173:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584538745.154175:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584538745.154183:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538745.154186:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538745.154187:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538745.154188:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538745.154189:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538745.154191:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584538745.154192:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584538745.154194:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20545 00000100:00080000:22.0:1584538745.154197:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584538745.154213:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538745.154214:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538745.154215:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538745.154216:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538745.154216:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538745.154218:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538745.154218:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538745.154220:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538745.216544:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584538745.216551:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584538745.216553:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584538745.216555:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584538745.216556:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584538745.216559:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584538745.216562:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584538745.216564:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538826 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584538745.216593:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078441776 new x1661492078441824 00000800:00020000:4.0:1584538751.916044:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584538751.916048:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584538751.916050:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538751.916475:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584538751.928539:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538745/real 1584538745] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538832 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584538751.928550:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584538751.928551:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584538751.928553:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584538751.928561:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538751.928563:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538751.928564:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538751.928565:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538751.928566:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538751.928568:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584538751.928569:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584538751.928571:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20551 00000100:00080000:22.0:1584538751.928574:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584538751.928592:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538751.928593:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538751.928594:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538751.928595:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538751.928596:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538751.928597:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538751.928599:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538751.928600:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538751.981001:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584538751.981008:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584538751.981010:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584538751.981012:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584538751.981014:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584538751.981016:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584538751.981018:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584538751.981020:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538832 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584538751.981051:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078441824 new x1661492078441856 00000800:00000100:0.0:1584538757.140077:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538757.140492:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584538757.152541:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538751/real 1584538751] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538838 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584538757.152550:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584538757.152551:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584538757.152553:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584538757.152558:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538757.152560:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538757.152561:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538757.152562:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538757.152563:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538757.152564:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584538757.152565:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584538757.152567:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20557 00000100:00080000:22.0:1584538757.152569:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584538757.152579:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538757.152581:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538757.152581:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538757.152583:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538757.152584:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538757.152585:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538757.152586:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538757.152587:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538757.214096:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584538757.214102:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584538757.214104:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584538757.214105:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584538757.214107:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584538757.214110:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584538757.214111:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584538757.214113:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538838 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584538757.214122:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078441856 new x1661492078441904 00000800:00000100:0.0:1584538763.140434:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538763.140855:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584538763.152906:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538757/real 1584538757] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538844 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584538763.152914:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584538763.152915:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584538763.152916:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584538763.152930:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538763.152931:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538763.152935:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538763.152936:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538763.152937:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538763.152938:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584538763.152939:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584538763.152940:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20563 00000100:00080000:22.0:1584538763.152942:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584538763.152948:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538763.152949:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538763.152953:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538763.152954:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538763.152955:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538763.152956:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538763.152956:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538763.152957:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538763.202795:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584538763.202799:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584538763.202801:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584538763.202801:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584538763.202803:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584538763.202805:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584538763.202806:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584538763.202808:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538844 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584538763.202816:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078441904 new x1661492078441968 00000800:00000100:0.0:1584538769.140310:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538769.140956:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584538769.153007:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538763/real 1584538763] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538850 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584538769.153015:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584538769.153016:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584538769.153017:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584538769.153023:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538769.153025:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538769.153025:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538769.153037:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538769.153038:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538769.153039:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584538769.153040:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584538769.153041:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20569 00000100:00080000:22.0:1584538769.153043:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584538769.153053:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538769.153054:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538769.153055:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538769.153056:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538769.153057:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538769.153058:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538769.153058:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538769.153060:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538769.179490:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584538769.179494:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584538769.179495:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584538769.179496:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584538769.179497:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584538769.179499:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584538769.179500:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584538769.179502:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538850 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584538769.179511:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078441968 new x1661492078442000 00000800:00020000:2.0:1584538775.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584538775.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584538775.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538775.916542:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584538775.928627:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538769/real 1584538769] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538856 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584538775.928635:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584538775.928637:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584538775.928638:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584538775.928644:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538775.928645:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538775.928646:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538775.928647:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538775.928648:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538775.928649:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584538775.928650:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584538775.928652:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20575 00000100:00080000:22.0:1584538775.928655:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584538775.928663:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538775.928665:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538775.928665:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538775.928666:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538775.928667:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538775.928668:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538775.928669:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538775.928670:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538775.986562:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584538775.986566:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584538775.986568:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584538775.986569:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584538775.986570:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584538775.986572:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584538775.986573:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584538775.986575:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538856 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584538775.986605:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078442000 new x1661492078442048 00000800:00000100:0.0:1584538781.140620:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538781.141084:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584538781.153154:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538775/real 1584538775] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538862 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584538781.153162:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584538781.153163:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584538781.153164:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584538781.153170:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538781.153172:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538781.153172:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538781.153173:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538781.153174:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538781.153176:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584538781.153177:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584538781.153178:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20581 00000100:00080000:22.0:1584538781.153181:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584538781.153189:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538781.153191:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538781.153191:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538781.153192:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538781.153193:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538781.153194:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538781.153195:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538781.153196:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584538781.180315:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584538781.180318:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584538781.180320:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584538781.180321:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584538781.180322:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584538781.180324:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584538781.180325:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584538781.180327:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538862 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584538781.180335:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078442048 new x1661492078442080 00000800:00000100:0.0:1584538787.140953:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538787.141360:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584538787.153409:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538781/real 1584538781] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538868 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584538787.153417:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584538787.153418:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584538787.153419:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584538787.153433:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538787.153435:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538787.153438:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538787.153440:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538787.153440:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538787.153442:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584538787.153442:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584538787.153444:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20587 00000100:00080000:22.0:1584538787.153446:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584538787.153451:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538787.153453:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538787.153453:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538787.153454:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538787.153455:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538787.153456:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538787.153457:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538787.153458:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538787.205182:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584538787.205186:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584538787.205188:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584538787.205189:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584538787.205190:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584538787.205192:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584538787.205194:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584538787.205195:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538868 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584538787.205225:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078442080 new x1661492078442160 00000800:00000100:0.0:1584538793.140664:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538793.141074:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584538793.153123:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538787/real 1584538787] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538874 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584538793.153131:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584538793.153132:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584538793.153133:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584538793.153139:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538793.153141:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538793.153142:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538793.153143:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538793.153144:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538793.153145:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584538793.153146:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584538793.153147:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20593 00000100:00080000:22.0:1584538793.153149:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584538793.153158:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538793.153159:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538793.153160:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538793.153161:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538793.153162:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538793.153163:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538793.153163:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538793.153164:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538793.195149:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584538793.195153:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584538793.195155:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584538793.195155:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584538793.195157:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584538793.195159:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584538793.195160:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584538793.195162:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538874 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584538793.195190:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078442160 new x1661492078442192 00000800:00020000:4.0:1584538799.916069:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584538799.916071:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584538799.916073:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538799.916479:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584538799.928535:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538793/real 1584538793] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538880 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584538799.928543:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584538799.928544:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584538799.928546:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584538799.928552:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538799.928553:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538799.928554:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538799.928555:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538799.928556:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538799.928557:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584538799.928558:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584538799.928560:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20599 00000100:00080000:22.0:1584538799.928562:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584538799.928570:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538799.928571:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538799.928572:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538799.928573:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538799.928574:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538799.928575:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538799.928575:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538799.928576:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538800.565659:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584538800.565664:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584538800.565665:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584538800.565666:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584538800.565668:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584538800.565670:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584538800.565671:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584538800.565673:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538880 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584538800.565702:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078442192 new x1661492078442224 00000800:00000100:0.0:1584538806.141009:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538806.141455:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584538806.153510:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538800/real 1584538800] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538887 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584538806.153525:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584538806.153526:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584538806.153528:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584538806.153535:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538806.153537:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538806.153538:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538806.153540:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538806.153540:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538806.153542:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584538806.153543:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584538806.153545:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20605 00000100:00080000:22.0:1584538806.153548:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584538806.153561:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538806.153563:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538806.153563:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538806.153565:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538806.153565:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538806.153567:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538806.153567:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538806.153569:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584538806.198061:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584538806.198068:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584538806.198070:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584538806.198071:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584538806.198073:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584538806.198075:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584538806.198076:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584538806.198078:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538887 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584538806.198086:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078442224 new x1661492078442272 00000800:00000100:0.0:1584538812.140942:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538812.141361:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584538812.153421:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538806/real 1584538806] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538893 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584538812.153432:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584538812.153434:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584538812.153436:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584538812.153461:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538812.153463:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538812.153471:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538812.153472:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538812.153473:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538812.153475:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584538812.153476:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584538812.153478:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20612 00000100:00080000:22.0:1584538812.153482:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584538812.153490:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538812.153491:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538812.153492:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538812.153493:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538812.153494:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538812.153495:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538812.153497:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538812.153499:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584538812.598986:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584538812.598993:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584538812.598996:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584538812.598997:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584538812.598999:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584538812.599001:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584538812.599003:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584538812.599005:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538893 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584538812.599016:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078442272 new x1661492078442336 00000800:00020000:4.0:1584538818.916063:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584538818.916066:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584538818.916068:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538818.916473:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584538818.928557:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538812/real 1584538812] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538899 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584538818.928577:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584538818.928578:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584538818.928579:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584538818.928586:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538818.928587:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538818.928588:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538818.928589:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538818.928590:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538818.928592:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584538818.928592:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584538818.928594:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20618 00000100:00080000:22.0:1584538818.928596:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584538818.928606:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538818.928607:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538818.928608:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538818.928609:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538818.928611:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538818.928612:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538818.928613:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538818.928614:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584538819.202545:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584538819.202551:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584538819.202553:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584538819.202554:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584538819.202556:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584538819.202558:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584538819.202559:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584538819.202561:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538899 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584538819.202591:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078442336 new x1661492078442384 00000800:00000100:0.0:1584538825.141083:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538825.141577:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584538825.153645:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538819/real 1584538819] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538906 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584538825.153653:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584538825.153654:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584538825.153655:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584538825.153661:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538825.153663:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538825.153663:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538825.153664:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538825.153665:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538825.153667:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584538825.153667:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584538825.153669:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20624 00000100:00080000:22.0:1584538825.153671:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584538825.153679:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538825.153681:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538825.153682:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538825.153683:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538825.153684:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538825.153685:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538825.153686:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538825.153687:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584538825.204631:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584538825.204635:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584538825.204636:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584538825.204637:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584538825.204638:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584538825.204640:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584538825.204642:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584538825.204643:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538906 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584538825.204673:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078442384 new x1661492078442416 00000800:00000100:0.0:1584538831.141213:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538831.141674:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584538831.153744:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538825/real 1584538825] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538912 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584538831.153753:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584538831.153755:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584538831.153757:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584538831.153765:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538831.153767:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538831.153768:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538831.153769:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538831.153770:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538831.153771:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584538831.153772:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584538831.153774:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20631 00000100:00080000:22.0:1584538831.153776:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584538831.153792:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538831.153793:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538831.153793:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538831.153795:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538831.153795:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538831.153797:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538831.153808:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538831.153810:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584538831.201092:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584538831.201099:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584538831.201101:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584538831.201103:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584538831.201105:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584538831.201107:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584538831.201109:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584538831.201111:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538912 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584538831.201122:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078442416 new x1661492078442448 00000800:00020000:6.0:1584538837.916064:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584538837.916067:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584538837.916069:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538837.916474:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584538837.928527:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538831/real 1584538831] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538918 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584538837.928536:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584538837.928537:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584538837.928538:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584538837.928554:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538837.928556:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538837.928559:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538837.928560:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538837.928561:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538837.928563:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584538837.928564:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584538837.928567:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20637 00000100:00080000:22.0:1584538837.928569:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584538837.928576:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538837.928577:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538837.928578:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538837.928579:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538837.928580:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538837.928581:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538837.928582:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538837.928583:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584538837.947876:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584538837.947882:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584538837.947884:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584538837.947885:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584538837.947887:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584538837.947889:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584538837.947890:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584538837.947892:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538918 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584538837.947922:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078442448 new x1661492078442528 00000800:00000100:2.0:1584538843.142386:0:4532:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538843.142871:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584538843.154961:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538837/real 1584538837] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538924 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584538843.154968:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584538843.154969:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584538843.154971:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584538843.154976:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538843.154978:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538843.154978:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538843.154980:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538843.154980:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538843.154983:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584538843.154983:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584538843.154985:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20643 00000100:00080000:22.0:1584538843.154987:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584538843.154995:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538843.154997:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538843.154997:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538843.154998:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538843.154999:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538843.155000:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538843.155001:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538843.155002:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584538843.162941:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584538843.162945:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584538843.162947:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584538843.162947:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584538843.162949:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584538843.162950:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584538843.162952:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584538843.162954:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538924 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584538843.162983:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078442528 new x1661492078442560 00000800:00000100:0.0:1584538849.141503:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538849.141941:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584538849.153990:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538843/real 1584538843] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538930 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584538849.153998:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584538849.153999:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584538849.154000:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584538849.154006:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538849.154008:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538849.154008:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538849.154011:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538849.154011:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538849.154013:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584538849.154013:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584538849.154015:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20649 00000100:00080000:22.0:1584538849.154017:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584538849.154025:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538849.154042:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538849.154043:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538849.154044:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538849.154045:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538849.154046:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538849.154046:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538849.154048:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584538849.218003:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584538849.218008:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584538849.218009:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584538849.218010:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584538849.218011:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584538849.218013:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584538849.218014:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584538849.218016:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538930 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584538849.218045:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078442560 new x1661492078442608 00000800:00000100:0.0:1584538855.141737:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538855.142218:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584538855.154282:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538849/real 1584538849] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538936 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584538855.154290:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584538855.154291:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584538855.154292:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584538855.154298:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538855.154300:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538855.154301:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538855.154302:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538855.154302:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538855.154304:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584538855.154304:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584538855.154306:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20655 00000100:00080000:22.0:1584538855.154308:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584538855.154316:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538855.154317:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538855.154318:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538855.154319:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538855.154319:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538855.154320:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538855.154321:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538855.154322:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584538855.205564:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584538855.205569:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584538855.205570:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584538855.205571:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584538855.205572:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584538855.205574:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584538855.205575:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584538855.205577:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538936 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584538855.205611:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078442608 new x1661492078442640 00000800:00020000:6.0:1584538861.916126:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584538861.916129:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584538861.916131:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538861.916688:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584538861.928779:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538855/real 1584538855] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538942 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584538861.928788:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584538861.928789:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584538861.928792:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584538861.928811:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538861.928813:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538861.928822:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538861.928823:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538861.928824:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538861.928826:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584538861.928826:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584538861.928828:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20661 00000100:00080000:22.0:1584538861.928831:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584538861.928837:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538861.928838:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538861.928839:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538861.928840:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538861.928841:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538861.928842:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584538861.928843:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584538861.928844:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584538861.936802:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584538861.936809:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584538861.936811:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584538861.936812:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584538861.936813:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584538861.936816:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584538861.936818:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584538861.936820:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538942 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584538861.936829:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078442640 new x1661492078442720 00000800:00000100:0.0:1584538867.141950:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538867.142379:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584538867.154452:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538861/real 1584538861] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538948 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584538867.154459:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584538867.154460:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584538867.154461:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584538867.154467:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538867.154469:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538867.154470:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538867.154471:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538867.154471:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538867.154473:0:4532:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584538867.154473:0:4532:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584538867.154475:0:4532:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20667 00000100:00080000:2.0:1584538867.154477:0:4532:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584538867.154487:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538867.154488:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538867.154489:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538867.154490:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538867.154491:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538867.154493:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538867.154504:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538867.154506:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538867.180095:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538867.180105:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538867.180108:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538867.180110:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538867.180113:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538867.180117:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538867.180121:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538867.180125:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538948 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584538867.180171:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078442720 new x1661492078442752 00000800:00000100:0.0:1584538873.140903:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538873.141365:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584538873.153435:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538867/real 1584538867] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538954 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584538873.153446:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584538873.153447:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584538873.153449:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584538873.153457:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538873.153459:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538873.153460:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538873.153461:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538873.153461:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538873.153463:0:4532:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584538873.153464:0:4532:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584538873.153466:0:4532:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20673 00000100:00080000:2.0:1584538873.153469:0:4532:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584538873.153483:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538873.153484:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538873.153485:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538873.153487:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538873.153488:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538873.153489:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538873.153490:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538873.153502:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538873.162023:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538873.162039:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538873.162041:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538873.162042:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538873.162045:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538873.162047:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538873.162048:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538873.162050:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538954 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584538873.162060:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078442752 new x1661492078442784 00000800:00000100:0.0:1584538879.140711:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538879.141115:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584538879.153185:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538873/real 1584538873] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538960 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584538879.153193:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584538879.153194:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584538879.153196:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584538879.153201:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538879.153203:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538879.153203:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538879.153204:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538879.153205:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538879.153206:0:4532:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584538879.153207:0:4532:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584538879.153209:0:4532:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20679 00000100:00080000:2.0:1584538879.153211:0:4532:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584538879.153220:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538879.153222:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538879.153223:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538879.153224:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538879.153225:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538879.153226:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538879.153237:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538879.153239:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538879.161366:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538879.161372:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538879.161374:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538879.161375:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538879.161377:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538879.161379:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538879.161381:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538879.161383:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538960 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584538879.161392:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078442784 new x1661492078442832 00000800:00020000:4.0:1584538885.916177:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584538885.916180:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584538885.916182:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538885.916595:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584538885.928648:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538879/real 1584538879] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538966 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584538885.928656:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584538885.928657:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584538885.928658:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584538885.928672:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538885.928674:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538885.928683:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538885.928684:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538885.928685:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538885.928686:0:4532:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584538885.928687:0:4532:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584538885.928689:0:4532:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20685 00000100:00080000:2.0:1584538885.928692:0:4532:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584538885.928698:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538885.928699:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538885.928700:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538885.928701:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538885.928702:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538885.928703:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584538885.928704:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584538885.928705:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538885.936562:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538885.936567:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538885.936569:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538885.936570:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538885.936571:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538885.936573:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538885.936575:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538885.936577:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538966 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584538885.936586:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078442832 new x1661492078442896 00000800:00000100:0.0:1584538891.140848:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538891.141277:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584538891.153327:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538885/real 1584538885] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538972 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584538891.153335:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584538891.153337:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584538891.153338:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584538891.153343:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538891.153345:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584538891.153346:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538891.153347:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584538891.153348:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538891.153349:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584538891.153350:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584538891.153351:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20691 00000100:00080000:4.0:1584538891.153354:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584538891.153364:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538891.153365:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584538891.153366:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538891.153367:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584538891.153368:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538891.153369:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584538891.153369:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538891.153370:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538891.161367:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538891.161371:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538891.161373:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538891.161373:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538891.161375:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538891.161377:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538891.161378:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538891.161380:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538972 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584538891.161408:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078442896 new x1661492078442928 00000800:00000100:0.0:1584538897.141325:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538897.141759:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584538897.153826:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538891/real 1584538891] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538978 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584538897.153834:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584538897.153835:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584538897.153838:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584538897.153843:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538897.153845:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584538897.153846:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538897.153847:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584538897.153847:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538897.153849:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584538897.153849:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584538897.153851:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20697 00000100:00080000:4.0:1584538897.153853:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584538897.153861:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538897.153862:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584538897.153863:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538897.153864:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584538897.153865:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538897.153866:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584538897.153867:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538897.153868:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538897.161636:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538897.161640:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538897.161642:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538897.161642:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538897.161645:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538897.161647:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538897.161648:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538897.161650:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538978 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584538897.161660:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078442928 new x1661492078442976 00000800:00000100:0.0:1584538903.141964:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538903.142398:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584538903.154466:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538897/real 1584538897] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538984 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584538903.154478:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584538903.154479:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584538903.154481:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584538903.154486:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538903.154488:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584538903.154489:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538903.154490:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584538903.154490:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538903.154492:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584538903.154492:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584538903.154494:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20703 00000100:00080000:4.0:1584538903.154496:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584538903.154504:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538903.154505:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584538903.154506:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538903.154507:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584538903.154507:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538903.154509:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584538903.154509:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584538903.154510:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538903.202067:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538903.202071:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538903.202073:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538903.202074:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538903.202075:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538903.202077:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538903.202078:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538903.202080:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538984 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584538903.202090:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078442976 new x1661492078443008 00000800:00020000:8.0:1584538909.916051:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584538909.916054:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584538909.916056:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584538909.916519:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584538909.928596:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538903/real 1584538903] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538990 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584538909.928604:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584538909.928606:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584538909.928607:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584538909.928623:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538909.928625:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538909.928632:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538909.928633:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538909.928633:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538909.928635:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584538909.928635:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584538909.928637:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20709 00000100:00080000:6.0:1584538909.928639:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584538909.928645:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538909.928648:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538909.928648:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538909.928650:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538909.928650:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538909.928651:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538909.928652:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538909.928653:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538909.973242:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538909.973246:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538909.973248:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538909.973249:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538909.973250:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538909.973252:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538909.973254:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538909.973256:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538990 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584538909.973265:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078443008 new x1661492078443088 00000800:00000100:0.0:1584538915.141367:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538915.141771:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584538915.153841:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538909/real 1584538909] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538996 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584538915.153848:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584538915.153849:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584538915.153851:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584538915.153856:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538915.153858:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538915.153859:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538915.153860:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538915.153861:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538915.153862:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584538915.153862:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584538915.153864:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20715 00000100:00080000:6.0:1584538915.153867:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584538915.153875:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538915.153876:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538915.153877:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538915.153878:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538915.153879:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538915.153880:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538915.153880:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538915.153882:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538915.193208:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538915.193212:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538915.193213:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538915.193214:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538915.193216:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538915.193218:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538915.193220:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538915.193222:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584538996 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584538915.193231:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078443088 new x1661492078443120 00000800:00000100:0.0:1584538921.141257:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538921.141695:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584538921.153747:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538915/real 1584538915] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539002 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584538921.153755:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584538921.153756:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584538921.153757:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584538921.153762:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538921.153763:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538921.153764:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538921.153765:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538921.153766:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538921.153767:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584538921.153768:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584538921.153770:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20721 00000100:00080000:6.0:1584538921.153772:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584538921.153780:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538921.153781:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538921.153782:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538921.153783:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538921.153784:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538921.153785:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538921.153785:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538921.153787:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538921.202411:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538921.202415:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538921.202416:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538921.202417:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538921.202419:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538921.202421:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538921.202422:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538921.202424:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539002 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584538921.202433:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078443120 new x1661492078443152 00000800:00000100:0.0:1584538927.141487:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538927.141940:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584538927.154055:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538921/real 1584538921] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539008 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584538927.154065:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584538927.154066:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584538927.154068:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584538927.154075:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538927.154077:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538927.154077:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538927.154079:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538927.154080:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538927.154082:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584538927.154082:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584538927.154084:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20727 00000100:00080000:6.0:1584538927.154087:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584538927.154099:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538927.154100:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538927.154101:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538927.154102:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538927.154103:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538927.154104:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538927.154105:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538927.154106:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538927.705870:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538927.705876:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538927.705878:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538927.705880:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538927.705882:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538927.705884:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538927.705886:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538927.705888:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539008 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584538927.705897:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078443152 new x1661492078443200 00000800:00000100:0.0:1584538933.141422:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538933.141904:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584538933.153977:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538927/real 1584538927] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539014 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584538933.153987:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584538933.153989:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584538933.153991:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584538933.154014:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538933.154015:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538933.154037:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538933.154038:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538933.154050:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538933.154052:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584538933.154053:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584538933.154055:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20733 00000100:00080000:6.0:1584538933.154059:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584538933.154067:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538933.154068:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538933.154069:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538933.154070:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538933.154071:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538933.154072:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538933.154073:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538933.154074:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584538933.194905:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584538933.194912:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584538933.194914:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584538933.194916:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584538933.194918:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584538933.194921:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584538933.194923:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584538933.194925:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539014 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584538933.194935:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078443200 new x1661492078443264 00000800:00000100:0.0:1584538939.140602:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538939.141010:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584538939.153066:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538933/real 1584538933] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539020 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584538939.153074:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584538939.153076:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584538939.153077:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584538939.153083:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538939.153085:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538939.153086:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538939.153087:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538939.153088:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538939.153089:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584538939.153090:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584538939.153092:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20739 00000100:00080000:6.0:1584538939.153095:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584538939.153105:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538939.153107:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538939.153107:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538939.153109:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538939.153109:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538939.153111:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538939.153111:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538939.153112:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538939.203408:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584538939.203415:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584538939.203417:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584538939.203418:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584538939.203420:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584538939.203423:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584538939.203424:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584538939.203426:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539020 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584538939.203455:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078443264 new x1661492078443312 00000800:00000100:0.0:1584538945.140667:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584538945.141054:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584538945.153122:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538939/real 1584538939] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539026 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584538945.153131:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584538945.153133:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584538945.153134:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584538945.153140:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538945.153142:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538945.153143:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538945.153144:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538945.153145:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538945.153146:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584538945.153147:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584538945.153149:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20745 00000100:00080000:6.0:1584538945.153151:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584538945.153161:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538945.153162:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538945.153163:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538945.153164:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538945.153165:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538945.153166:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538945.153167:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538945.153168:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000800:00000400:2.0:1584538945.916081:0:25711:0:(o2iblnd_cb.c:3381:kiblnd_check_conns()) Timed out tx for 10.10.2.22@o2ib: 0 seconds 00000400:00000100:2.0:1584538945.927253:0:25711:0:(lib-msg.c:705:lnet_health_check()) msg 0@<0:0>->10.10.2.22@o2ib exceeded retry count 0 00000100:00080000:18.0:1584538945.927263:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:18.0:1584538945.927265:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0003_UUID on 10.10.2.22@o2ib failed (-110) 00000100:00080000:6.0:1584538970.201080:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538970.201084:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538970.201088:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538970.201089:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538970.201089:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:6.0:1584538970.201091:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584538970.201092:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584538970.201094:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20751 00000100:00080000:6.0:1584538970.201096:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 20112 00000100:00080000:6.0:1584538970.201098:0:42:0:(import.c:596:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0003 (at 10.10.2.21@o2ib) 00000100:00080000:6.0:1584538970.201099:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.21@o2ib/10.10.2.21@o2ib 00000100:00080000:6.0:1584538970.201106:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538970.201108:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538970.201109:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538970.201110:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538970.201111:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538970.201112:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538970.201113:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538970.201114:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538970.201309:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:18.0:1584538970.201311:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0003_UUID on 10.10.2.21@o2ib failed (-19) 00000100:00080000:6.0:1584538995.225078:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538995.225081:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538995.225087:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538995.225088:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538995.225089:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:6.0:1584538995.225091:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584538995.225092:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584538995.225096:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20751 00000100:00080000:6.0:1584538995.225098:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 20776 00000100:00080000:6.0:1584538995.225099:0:42:0:(import.c:570:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: tried all connections, increasing latency to 6s 00000100:00080000:6.0:1584538995.225102:0:42:0:(import.c:596:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0003 (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584538995.225103:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584538995.225112:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538995.225113:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538995.225114:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538995.225115:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538995.225116:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538995.225117:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584538995.225118:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584538995.225119:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584538995.225268:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584538995.225275:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584538995.225278:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584538995.225279:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584538995.225281:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584538995.225284:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584538995.225285:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584538995.225287:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539026 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584538995.225298:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078443312 new x1661492078443488 00000800:00020000:22.0:1584539001.916158:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:22.0:1584539001.916161:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:22.0:1584539001.916163:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539001.916661:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539001.928735:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584538995/real 1584538995] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539082 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539001.972540:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539001.972541:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539001.972543:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539001.972549:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539001.972551:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539001.972552:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539001.972553:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539001.972554:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539001.972556:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539001.972556:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539001.972558:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20801 00000100:00080000:6.0:1584539001.972561:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539001.972571:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539001.972572:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539001.972573:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539001.972574:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539001.972575:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539001.972576:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539001.972577:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539001.972578:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539002.021979:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539002.021984:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539002.021986:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539002.021987:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539002.021990:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539002.021992:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539002.021994:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539002.021996:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539082 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539002.022026:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078443488 new x1661492078443536 00000800:00000100:0.0:1584539007.140858:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539007.141370:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539007.153436:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539002/real 1584539002] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539089 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539007.153444:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539007.153446:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539007.153447:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539007.153453:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539007.153455:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539007.153456:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539007.153457:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539007.153457:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539007.153459:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539007.153459:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539007.153461:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20807 00000100:00080000:6.0:1584539007.153463:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539007.153472:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539007.153473:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539007.153474:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539007.153475:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539007.153475:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539007.153476:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539007.153478:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539007.153479:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539007.220235:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539007.220239:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539007.220240:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539007.220241:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539007.220243:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539007.220245:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539007.220246:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539007.220248:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539089 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539007.220277:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078443536 new x1661492078443568 00000800:00000100:0.0:1584539013.140993:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539013.141378:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539013.153445:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539007/real 1584539007] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539094 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539013.153453:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539013.153454:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539013.153455:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539013.153461:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539013.153463:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539013.153463:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539013.153465:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539013.153465:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539013.153466:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539013.153467:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539013.153469:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20813 00000100:00080000:6.0:1584539013.153471:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539013.153480:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539013.153481:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539013.153482:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539013.153483:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539013.153484:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539013.153485:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539013.153486:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539013.153487:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539013.209910:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539013.209914:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539013.209916:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539013.209916:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539013.209918:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539013.209920:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539013.209921:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539013.209923:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539094 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539013.209952:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078443568 new x1661492078443616 00000800:00000100:0.0:1584539019.141061:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539019.141493:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539019.153560:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539013/real 1584539013] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539100 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539019.153568:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539019.153569:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539019.153570:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539019.153584:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539019.153586:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539019.153594:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539019.153596:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539019.153596:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539019.153598:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539019.153598:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539019.153600:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20819 00000100:00080000:6.0:1584539019.153602:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539019.153607:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539019.153609:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539019.153610:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539019.153611:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539019.153612:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539019.153613:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539019.153614:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539019.153615:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539019.218579:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539019.218583:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539019.218585:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539019.218585:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539019.218588:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539019.218590:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539019.218591:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539019.218593:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539100 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539019.218604:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078443616 new x1661492078443680 00000800:00020000:10.0:1584539025.916107:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:10.0:1584539025.916111:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:10.0:1584539025.916112:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539025.916589:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539025.928658:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539019/real 1584539019] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539106 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539025.928666:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539025.928668:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539025.928669:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539025.928674:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539025.928676:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539025.928677:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539025.928678:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539025.928678:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539025.928680:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539025.928680:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539025.928682:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20825 00000100:00080000:8.0:1584539025.928684:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539025.928693:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539025.928694:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539025.928695:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539025.928696:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539025.928697:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539025.928698:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539025.928699:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539025.928700:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539025.963801:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539025.963805:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539025.963806:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539025.963807:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539025.963808:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539025.963810:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539025.963812:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539025.963814:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539106 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539025.963842:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078443680 new x1661492078443712 00000800:00000100:0.0:1584539031.141262:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539031.141791:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539031.153860:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539025/real 1584539025] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539112 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539031.153869:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539031.179223:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539031.179225:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539031.179230:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539031.179231:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539031.179232:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539031.179233:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539031.179234:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539031.179235:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539031.179236:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539031.179237:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20831 00000100:00080000:8.0:1584539031.179239:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539031.179248:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539031.179250:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539031.179250:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539031.179251:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539031.179252:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539031.179253:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539031.179254:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539031.179255:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539031.225359:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539031.225363:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539031.225365:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539031.225366:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539031.225367:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539031.225369:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539031.225370:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539031.243169:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539112 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539031.243201:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078443712 new x1661492078443760 00000800:00000100:0.0:1584539037.141393:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539037.141894:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539037.153969:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539031/real 1584539031] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539118 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539037.153976:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539037.153977:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539037.153979:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539037.153983:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539037.153985:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539037.153986:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539037.153987:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539037.153988:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539037.153989:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539037.153989:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539037.153991:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20837 00000100:00080000:8.0:1584539037.153993:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539037.154001:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539037.154002:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539037.154003:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539037.154004:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539037.154004:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539037.154005:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539037.154007:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539037.154008:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539037.991449:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539037.991453:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539037.991454:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539037.991455:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539037.991457:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539037.991459:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539037.991460:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539037.991462:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539118 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539037.991472:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078443760 new x1661492078443792 00000800:00000100:0.0:1584539043.141491:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539043.141941:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539043.154039:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539037/real 1584539037] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539124 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539043.154048:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539043.154049:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539043.154050:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539043.154065:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539043.154067:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539043.154069:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539043.154082:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539043.154082:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539043.154084:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539043.154084:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539043.154086:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20843 00000100:00080000:8.0:1584539043.154088:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539043.154106:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539043.154107:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539043.154107:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539043.154109:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539043.154110:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539043.154112:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539043.154112:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539043.154113:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539043.198232:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584539043.198237:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584539043.198239:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584539043.198240:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584539043.198242:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584539043.198244:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584539043.198245:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584539043.198247:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539124 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539043.198258:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078443792 new x1661492078443872 00000800:00020000:4.0:1584539049.916094:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584539049.916097:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584539049.916099:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539049.916521:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539049.928619:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539043/real 1584539043] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539130 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539049.928640:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539049.928642:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539049.928643:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539049.928650:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539049.928653:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539049.928654:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539049.928655:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539049.928656:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539049.928657:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539049.928658:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539049.928660:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20849 00000100:00080000:8.0:1584539049.928663:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539049.928678:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539049.928680:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539049.928681:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539049.928683:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539049.928683:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539049.928685:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539049.928685:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539049.928687:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539049.967052:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584539049.967059:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584539049.967061:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584539049.967062:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584539049.967065:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584539049.967067:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584539049.967068:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584539049.967070:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539130 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539049.967080:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078443872 new x1661492078443904 00000800:00000100:0.0:1584539055.140755:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539055.141235:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539055.153323:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539049/real 1584539049] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539136 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539055.153332:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539055.153334:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539055.153335:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539055.153343:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539055.153345:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539055.153346:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539055.153347:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539055.153348:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539055.153350:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539055.153351:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539055.153353:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20855 00000100:00080000:8.0:1584539055.153356:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539055.153373:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539055.153374:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539055.153375:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539055.153376:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539055.153377:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539055.153378:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539055.153389:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539055.153391:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539055.193473:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584539055.193480:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584539055.193482:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584539055.193483:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584539055.193485:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584539055.193487:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584539055.193489:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584539055.193491:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539136 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539055.193525:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078443904 new x1661492078443936 00000800:00000100:0.0:1584539061.140546:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539061.141079:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539061.153130:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539055/real 1584539055] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539142 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539061.153139:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539061.153140:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539061.153142:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539061.153147:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539061.153149:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539061.153150:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539061.153152:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539061.153152:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539061.153154:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539061.153155:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539061.153157:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20861 00000100:00080000:8.0:1584539061.153160:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539061.153170:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539061.153172:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539061.153172:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539061.153174:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539061.153174:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539061.153176:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539061.153176:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539061.153178:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539061.196983:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584539061.196990:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584539061.196992:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584539061.196993:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584539061.196994:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584539061.196997:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584539061.196998:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584539061.197000:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539142 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539061.197042:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078443936 new x1661492078443984 00000800:00000100:0.0:1584539067.140717:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539067.141215:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539067.153290:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539061/real 1584539061] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539148 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539067.153298:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539067.153299:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539067.153300:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539067.153328:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539067.153330:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539067.153333:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539067.153345:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539067.153346:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539067.153348:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539067.153349:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539067.153350:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20867 00000100:00080000:8.0:1584539067.153352:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539067.153370:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539067.153372:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539067.153372:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539067.153374:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539067.153374:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539067.153375:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539067.153376:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539067.153377:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539067.161329:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584539067.161335:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584539067.161337:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584539067.161338:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584539067.161340:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584539067.161342:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584539067.161344:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584539067.161346:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539148 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539067.161374:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078443984 new x1661492078444048 00000800:00020000:4.0:1584539073.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584539073.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584539073.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539073.916530:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539073.928617:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539067/real 1584539067] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539154 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539073.928625:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539073.928626:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539073.928627:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539073.928633:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539073.928635:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539073.928636:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539073.928638:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539073.928639:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539073.928640:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539073.928641:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539073.928643:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20873 00000100:00080000:8.0:1584539073.928645:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539073.928654:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539073.928656:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539073.928656:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539073.928658:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539073.928658:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539073.928659:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539073.928660:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539073.928661:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539073.936674:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584539073.936679:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584539073.936680:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584539073.936681:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584539073.936683:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584539073.936685:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584539073.936686:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584539073.936688:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539154 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539073.936721:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078444048 new x1661492078444096 00000800:00000100:0.0:1584539079.140920:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539079.141391:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539079.153475:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539073/real 1584539073] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539160 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539079.153483:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539079.153484:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539079.153485:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539079.153491:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539079.153493:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539079.153494:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539079.153495:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539079.153495:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539079.153497:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539079.153497:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539079.153499:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20879 00000100:00080000:8.0:1584539079.153501:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539079.153509:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539079.153510:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539079.153511:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539079.153512:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539079.153513:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539079.153514:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539079.153514:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539079.153527:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539079.161532:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584539079.161536:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584539079.161537:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584539079.161538:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584539079.161540:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584539079.161542:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584539079.161543:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584539079.161545:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539160 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539079.161573:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078444096 new x1661492078444128 00000800:00000100:0.0:1584539085.140941:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539085.141375:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539085.153431:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539079/real 1584539079] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539166 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539085.153438:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539085.153440:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539085.153442:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539085.153446:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539085.153448:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539085.153449:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539085.153450:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539085.153450:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539085.153452:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539085.153452:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539085.153454:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20885 00000100:00080000:8.0:1584539085.153456:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539085.153465:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539085.153466:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539085.153467:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539085.153468:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539085.153468:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539085.153469:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539085.153470:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539085.153471:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539085.161523:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584539085.161527:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584539085.161529:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584539085.161529:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584539085.161532:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584539085.161533:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584539085.161535:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584539085.161537:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539166 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539085.161565:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078444128 new x1661492078444160 00000800:00000100:0.0:1584539091.141109:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539091.141609:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539091.153676:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539085/real 1584539085] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539172 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539091.153687:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539091.153688:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539091.153690:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539091.153704:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539091.153705:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539091.153719:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539091.153720:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539091.153721:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539091.153722:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539091.153723:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539091.153724:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20891 00000100:00080000:8.0:1584539091.153726:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539091.153743:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539091.153744:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539091.153745:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539091.153746:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539091.153747:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539091.153748:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539091.153748:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539091.153750:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539091.161800:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584539091.161805:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584539091.161806:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584539091.161807:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584539091.161808:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584539091.161810:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584539091.161812:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584539091.161814:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539172 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539091.161823:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078444160 new x1661492078444240 00000800:00020000:4.0:1584539097.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584539097.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584539097.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539097.916514:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539097.928595:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539091/real 1584539091] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539178 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539097.928603:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539097.928604:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539097.928605:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539097.928611:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539097.928613:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539097.928614:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539097.928615:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539097.928615:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539097.928617:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539097.928617:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539097.928619:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20897 00000100:00080000:8.0:1584539097.928621:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539097.928629:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539097.928630:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539097.928631:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539097.928632:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539097.928633:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539097.928634:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539097.928636:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539097.928637:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539097.936402:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584539097.936406:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584539097.936408:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584539097.936409:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584539097.936410:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584539097.936412:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584539097.936413:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584539097.936415:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539178 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539097.936444:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078444240 new x1661492078444272 00000800:00000100:0.0:1584539103.141283:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539103.141732:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539103.153810:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539097/real 1584539097] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539184 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539103.153819:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539103.153820:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539103.153821:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539103.153828:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539103.153830:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539103.153831:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539103.153832:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539103.153833:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539103.153834:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539103.153835:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539103.153837:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20903 00000100:00080000:8.0:1584539103.153839:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539103.153851:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539103.153852:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539103.153853:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539103.153854:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539103.153856:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539103.153868:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539103.153869:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539103.153870:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539103.162003:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584539103.162009:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584539103.162011:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584539103.162012:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584539103.162015:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584539103.162017:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584539103.162018:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584539103.162020:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539184 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539103.162054:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078444272 new x1661492078444320 00000800:00000100:0.0:1584539109.141303:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539109.141714:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539109.153760:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539103/real 1584539103] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539190 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539109.153769:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539109.153770:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539109.153771:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539109.153777:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539109.153779:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539109.153780:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539109.153781:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539109.153782:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539109.153783:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539109.153784:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539109.153786:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20909 00000100:00080000:8.0:1584539109.153788:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539109.153801:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539109.153802:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539109.153804:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539109.153805:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539109.153806:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539109.153807:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539109.153819:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539109.153820:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539109.162160:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584539109.162167:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584539109.162169:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584539109.162170:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584539109.162172:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584539109.162174:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584539109.162176:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584539109.162178:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539190 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539109.162206:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078444320 new x1661492078444352 00000800:00000100:0.0:1584539115.140611:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539115.141049:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539115.153128:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539109/real 1584539109] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539196 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539115.153137:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539115.153138:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539115.153139:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539115.153159:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539115.153160:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539115.153167:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539115.153168:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539115.153169:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539115.153171:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539115.153171:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539115.153173:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20915 00000100:00080000:8.0:1584539115.153176:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539115.153184:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539115.153186:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539115.153187:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539115.153188:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539115.153189:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539115.153190:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539115.153190:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539115.153192:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539115.160974:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584539115.160981:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584539115.160983:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584539115.160984:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584539115.160986:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584539115.160988:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584539115.160990:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584539115.160992:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539196 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539115.161021:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078444352 new x1661492078444416 00000800:00020000:2.0:1584539121.916073:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584539121.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584539121.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539121.916519:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539121.928577:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539115/real 1584539115] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539202 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539121.928585:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539121.928586:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539121.928587:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539121.928592:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539121.928594:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539121.928595:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539121.928596:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539121.928596:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539121.928598:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539121.928598:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539121.928601:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20921 00000100:00080000:8.0:1584539121.928603:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539121.928611:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539121.928613:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539121.928613:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539121.928615:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539121.928615:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539121.928616:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539121.928617:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539121.928618:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539121.936509:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584539121.936514:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584539121.936515:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584539121.936516:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584539121.936518:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584539121.936520:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584539121.936522:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584539121.936524:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539202 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539121.936552:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078444416 new x1661492078444464 00000800:00000100:0.0:1584539127.140835:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539127.141321:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539127.153388:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539121/real 1584539121] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539208 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539127.153396:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539127.153397:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539127.153398:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539127.153404:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539127.153406:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539127.153406:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539127.153407:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539127.153408:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539127.153411:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539127.153411:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539127.153413:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20927 00000100:00080000:8.0:1584539127.153415:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539127.153423:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539127.153425:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539127.153425:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539127.153426:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539127.153427:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539127.153428:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539127.153429:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539127.153430:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539127.161136:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584539127.161140:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584539127.161142:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584539127.161143:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584539127.161144:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584539127.161146:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584539127.161147:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584539127.161149:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539208 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539127.161159:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078444464 new x1661492078444496 00000800:00000100:0.0:1584539133.140950:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539133.141378:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539133.153428:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539127/real 1584539127] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539214 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539133.153435:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539133.153437:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539133.153438:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539133.153443:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539133.153445:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539133.153446:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539133.153448:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539133.153448:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539133.153450:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539133.153450:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539133.153452:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20933 00000100:00080000:8.0:1584539133.153454:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539133.153463:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539133.153464:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539133.153465:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539133.153466:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539133.153466:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539133.153467:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539133.153468:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539133.153469:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539133.161778:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584539133.161782:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584539133.161784:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584539133.161785:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584539133.161786:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584539133.161788:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584539133.161789:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584539133.161791:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539214 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539133.161820:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078444496 new x1661492078444544 00000800:00000100:0.0:1584539139.141039:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539139.141681:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539139.153732:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539133/real 1584539133] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539220 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539139.153739:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539139.153740:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539139.153742:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539139.153755:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539139.153757:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539139.153763:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539139.153765:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539139.153765:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539139.153767:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539139.153767:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539139.153769:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20939 00000100:00080000:8.0:1584539139.153771:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539139.153776:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539139.153777:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539139.153778:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539139.153779:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539139.153780:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539139.153781:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539139.153781:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539139.153782:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539139.206524:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584539139.206528:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584539139.206529:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584539139.206530:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584539139.206531:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584539139.206533:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584539139.206535:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584539139.206536:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539220 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539139.206566:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078444544 new x1661492078444608 00000800:00020000:4.0:1584539145.916051:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584539145.916053:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584539145.916055:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539145.916445:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539145.928495:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539139/real 1584539139] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539226 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539145.928502:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539145.928503:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539145.928505:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539145.928510:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539145.928512:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539145.928512:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539145.928513:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539145.928514:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539145.928515:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539145.928516:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539145.928518:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20945 00000100:00080000:8.0:1584539145.928520:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539145.928528:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539145.928529:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539145.928530:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539145.928531:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539145.928532:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539145.928533:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539145.928533:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539145.928534:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539145.966961:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584539145.966965:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584539145.966967:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584539145.966968:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584539145.966969:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584539145.966971:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584539145.966972:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584539145.966974:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539226 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539145.967003:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078444608 new x1661492078444640 00000800:00000100:0.0:1584539151.141360:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539151.141788:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539151.153837:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539145/real 1584539145] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539232 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539151.153849:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539151.153850:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539151.153851:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539151.153857:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539151.153859:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539151.153860:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539151.153861:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539151.153861:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539151.153863:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539151.153863:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539151.153865:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20951 00000100:00080000:8.0:1584539151.153867:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539151.153875:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539151.153877:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539151.153877:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539151.153878:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539151.153879:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539151.153880:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539151.153881:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539151.153882:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539151.193424:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584539151.193428:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584539151.193429:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584539151.193430:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584539151.193431:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584539151.193433:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584539151.193435:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584539151.193436:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539232 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539151.193466:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078444640 new x1661492078444688 00000800:00000100:0.0:1584539157.141253:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539157.141654:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539157.153703:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539151/real 1584539151] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539238 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539157.153710:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539157.153711:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539157.153713:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539157.153717:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539157.153718:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539157.153719:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539157.153720:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539157.153720:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539157.153722:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539157.153722:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539157.153724:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20957 00000100:00080000:8.0:1584539157.153726:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539157.153734:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539157.153736:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539157.153736:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539157.153737:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539157.153738:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539157.153739:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539157.153741:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539157.153742:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539157.189693:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584539157.189697:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584539157.189699:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584539157.189699:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584539157.189701:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584539157.189702:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584539157.189704:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584539157.189706:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539238 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539157.189735:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078444688 new x1661492078444720 00000800:00000100:0.0:1584539163.141353:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539163.141885:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539163.153936:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539157/real 1584539157] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539244 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539163.153945:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539163.153946:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539163.153948:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539163.153961:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539163.153963:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539163.153972:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539163.153973:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539163.153974:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539163.153976:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539163.153977:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539163.153979:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20963 00000100:00080000:8.0:1584539163.153981:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539163.153988:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539163.153989:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539163.153990:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539163.153991:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539163.153994:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539163.153995:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539163.153995:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539163.153997:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539163.186584:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584539163.186590:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584539163.186592:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584539163.186594:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584539163.186595:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584539163.186597:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584539163.186599:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584539163.186601:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539244 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539163.186630:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078444720 new x1661492078444800 00000800:00020000:6.0:1584539169.916041:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584539169.916045:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584539169.916048:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539169.916476:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539169.928527:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539163/real 1584539163] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539250 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539169.928540:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539169.928541:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539169.928543:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539169.928552:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539169.928554:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539169.928555:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539169.928557:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539169.928557:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539169.928559:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539169.928560:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539169.928562:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20969 00000100:00080000:10.0:1584539169.928566:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539169.928585:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539169.928586:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539169.928587:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539169.928588:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539169.928589:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539169.928591:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539169.928592:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539169.928593:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539169.979996:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584539169.980003:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584539169.980006:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584539169.980007:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584539169.980009:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584539169.980011:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584539169.980013:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584539169.980015:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539250 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539169.980023:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078444800 new x1661492078444832 00000800:00000100:0.0:1584539175.140729:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539175.141157:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539175.153211:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539169/real 1584539169] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539256 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539175.153222:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539175.153224:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539175.153226:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539175.153233:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539175.153235:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539175.153236:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539175.153237:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539175.153238:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539175.153240:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539175.153240:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539175.153242:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20975 00000100:00080000:10.0:1584539175.153246:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539175.153262:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539175.153264:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539175.153264:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539175.153267:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539175.153268:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539175.153269:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539175.153270:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539175.153271:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539175.195647:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584539175.195654:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584539175.195657:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584539175.195658:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584539175.195660:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584539175.195663:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584539175.195664:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584539175.195666:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539256 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539175.195675:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078444832 new x1661492078444864 00000800:00000100:0.0:1584539181.140747:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539181.141163:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539181.153215:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539175/real 1584539175] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539262 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539181.153224:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539181.153225:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539181.153227:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539181.153234:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539181.153236:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539181.153237:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539181.153238:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539181.153239:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539181.153240:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539181.153241:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539181.153242:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20981 00000100:00080000:10.0:1584539181.153245:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539181.153257:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539181.153259:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539181.153260:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539181.153261:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539181.153262:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539181.153263:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539181.153264:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539181.153265:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539181.202376:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584539181.202382:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584539181.202385:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584539181.202386:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584539181.202387:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584539181.202390:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584539181.202391:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584539181.202393:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539262 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539181.202402:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078444864 new x1661492078444912 00000800:00000100:0.0:1584539187.140793:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539187.141208:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539187.153258:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539181/real 1584539181] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539268 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539187.153266:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539187.153268:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539187.153269:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539187.153288:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539187.153290:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539187.153299:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539187.153300:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539187.153301:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539187.153302:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539187.153303:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539187.153305:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20987 00000100:00080000:10.0:1584539187.153308:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539187.153315:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539187.153316:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539187.153317:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539187.153319:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539187.153319:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539187.153321:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539187.153321:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539187.153322:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539187.192836:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584539187.192842:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584539187.192844:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584539187.192845:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584539187.192847:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584539187.192849:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584539187.192851:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584539187.192853:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539268 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539187.192881:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078444912 new x1661492078444976 00000800:00020000:6.0:1584539193.916041:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584539193.916044:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584539193.916046:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539193.916508:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539193.928562:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539187/real 1584539187] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539274 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539193.928570:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539193.928571:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539193.928573:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539193.928578:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539193.928580:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539193.928581:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539193.928582:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539193.928583:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539193.928584:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539193.928586:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539193.928588:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20993 00000100:00080000:10.0:1584539193.928590:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539193.928600:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539193.928601:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539193.928602:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539193.928603:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539193.928604:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539193.928605:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539193.928606:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539193.928607:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539193.965749:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584539193.965755:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584539193.965756:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584539193.965757:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584539193.965759:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584539193.965761:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584539193.965763:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584539193.965764:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539274 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539193.965794:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078444976 new x1661492078445024 00000800:00000100:0.0:1584539199.140937:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539199.141383:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539199.153432:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539193/real 1584539193] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539199.153440:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539199.153441:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539199.153442:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539199.153448:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539199.153450:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539199.153450:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539199.153452:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539199.153453:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539199.153455:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539199.153455:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539199.153457:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20999 00000100:00080000:10.0:1584539199.153459:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539199.153468:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539199.153469:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539199.153470:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539199.153471:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539199.153472:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539199.153473:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539199.153473:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539199.153475:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539199.189484:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584539199.189489:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584539199.189491:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584539199.189491:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584539199.189493:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584539199.189495:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584539199.189496:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584539199.189498:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539199.189506:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078445024 new x1661492078445056 00000800:00000100:0.0:1584539205.141173:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539205.141580:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539205.153630:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539199/real 1584539199] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539205.153637:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539205.153638:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539205.153639:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539205.153645:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539205.153647:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539205.153648:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539205.153650:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539205.153650:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539205.153652:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539205.153652:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539205.153654:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21005 00000100:00080000:10.0:1584539205.153656:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539205.153664:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539205.153665:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539205.153665:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539205.153667:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539205.153667:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539205.153668:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539205.153669:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539205.153670:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539205.207339:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584539205.207343:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584539205.207345:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584539205.207345:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584539205.207347:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584539205.207349:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584539205.207350:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584539205.207352:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539205.207360:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078445056 new x1661492078445088 00000800:00000100:2.0:1584539211.141202:0:4532:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539211.141845:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539211.153898:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539205/real 1584539205] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539211.153906:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539211.153907:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539211.153908:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539211.153925:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539211.153926:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539211.153929:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539211.153930:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539211.153931:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539211.153932:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539211.153933:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539211.153935:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21011 00000100:00080000:10.0:1584539211.153936:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539211.153942:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539211.153943:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539211.153944:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539211.153945:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539211.153945:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539211.153947:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539211.153948:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539211.153949:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539211.190175:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584539211.190179:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584539211.190181:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584539211.190181:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584539211.190183:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584539211.190185:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584539211.190186:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584539211.190188:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539211.190196:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078445088 new x1661492078445168 00000800:00020000:4.0:1584539217.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584539217.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584539217.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539217.916490:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584539217.928568:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539211/real 1584539211] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584539217.928576:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584539217.928577:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584539217.928578:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584539217.928584:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539217.928586:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539217.928588:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539217.928589:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539217.928589:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539217.928591:0:4532:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584539217.928591:0:4532:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584539217.928593:0:4532:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21017 00000100:00080000:2.0:1584539217.928595:0:4532:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584539217.928605:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539217.928606:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539217.928606:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539217.928608:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539217.928608:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539217.928609:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539217.928610:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539217.928611:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539217.988221:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584539217.988226:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584539217.988228:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584539217.988228:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584539217.988230:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584539217.988231:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584539217.988233:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584539217.988235:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584539217.988244:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078445168 new x1661492078445200 00000800:00000100:0.0:1584539223.141318:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539223.141738:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584539223.153798:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539217/real 1584539217] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584539223.153808:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584539223.153809:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584539223.153811:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584539223.153817:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539223.153819:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539223.153820:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539223.153821:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539223.153822:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539223.153823:0:4532:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584539223.153824:0:4532:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584539223.153826:0:4532:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21023 00000100:00080000:2.0:1584539223.153829:0:4532:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584539223.153841:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539223.153843:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539223.153843:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539223.153845:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539223.153845:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539223.153847:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539223.153847:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539223.153849:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539223.183899:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584539223.183905:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584539223.183908:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584539223.183909:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584539223.183910:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584539223.183913:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584539223.183914:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584539223.183916:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584539223.183925:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078445200 new x1661492078445248 00000800:00000100:0.0:1584539229.141384:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539229.141857:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584539229.153931:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539223/real 1584539223] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584539229.153940:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584539229.153942:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584539229.153944:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584539229.153950:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539229.153952:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539229.153953:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539229.153954:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539229.153955:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539229.153956:0:4532:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584539229.153957:0:4532:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584539229.153959:0:4532:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21029 00000100:00080000:2.0:1584539229.153962:0:4532:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584539229.153974:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539229.153975:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539229.153976:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539229.153977:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539229.153978:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539229.153979:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539229.153980:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539229.153981:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539229.183910:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584539229.183916:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584539229.183918:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584539229.183919:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584539229.183921:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584539229.183923:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584539229.183925:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584539229.183926:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584539229.183936:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078445248 new x1661492078445280 00000800:00000100:0.0:1584539235.140616:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539235.141036:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584539235.153118:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539229/real 1584539229] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584539235.153128:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584539235.153129:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584539235.153130:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584539235.153151:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539235.153153:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539235.153162:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539235.153164:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539235.153164:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539235.153166:0:4532:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584539235.153167:0:4532:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584539235.153169:0:4532:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21035 00000100:00080000:2.0:1584539235.153171:0:4532:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584539235.153178:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539235.153179:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539235.153180:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539235.153181:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539235.153182:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539235.153183:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539235.153184:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584539235.153185:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539235.193936:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584539235.193943:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584539235.193946:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584539235.193946:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584539235.193948:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584539235.193951:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584539235.193952:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584539235.193954:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584539235.193964:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078445280 new x1661492078445344 00000800:00020000:4.0:1584539241.916070:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584539241.938885:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584539241.963058:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539241.963443:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584539241.975520:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539235/real 1584539235] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539322 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584539241.975528:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584539241.975529:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584539241.975530:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584539241.975537:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539241.975539:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539241.975539:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539241.975544:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539241.975545:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539241.975546:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584539241.975547:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584539241.975548:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21041 00000100:00080000:4.0:1584539241.975551:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584539241.975561:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539241.975562:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539241.975563:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539241.975564:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539241.975565:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539241.975566:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539241.975567:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539241.975568:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539242.005062:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584539242.005067:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584539242.005069:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584539242.005070:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584539242.005071:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584539242.005073:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584539242.005075:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584539242.005077:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539322 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584539242.005087:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078445344 new x1661492078445392 00000800:00000100:0.0:1584539247.140814:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539247.141317:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584539247.153386:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539242/real 1584539242] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539329 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584539247.153394:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584539247.153395:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584539247.153396:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584539247.153402:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539247.153404:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539247.153405:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539247.153406:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539247.153407:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539247.153408:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584539247.153409:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584539247.153410:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21047 00000100:00080000:4.0:1584539247.153412:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584539247.153420:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539247.153422:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539247.153422:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539247.153423:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539247.153424:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539247.153425:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539247.153426:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539247.153427:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539247.203052:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584539247.203056:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584539247.203058:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584539247.203058:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584539247.203060:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584539247.203062:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584539247.203063:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584539247.203065:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539329 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584539247.203075:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078445392 new x1661492078445424 00000800:00000100:0.0:1584539253.140925:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539253.141337:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584539253.153408:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539247/real 1584539247] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539334 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584539253.153416:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584539253.153418:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584539253.153420:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584539253.153426:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539253.153428:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539253.153428:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539253.153430:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539253.153430:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539253.153432:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584539253.153432:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584539253.153434:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21053 00000100:00080000:4.0:1584539253.153436:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584539253.153445:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539253.153447:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539253.153447:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539253.153449:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539253.153449:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539253.153450:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539253.153451:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539253.153452:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539253.184634:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584539253.184638:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584539253.184640:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584539253.184641:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584539253.184642:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584539253.184644:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584539253.184646:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584539253.184648:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539334 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584539253.184677:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078445424 new x1661492078445472 00000800:00000100:0.0:1584539259.141243:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539259.141665:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584539259.153736:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539253/real 1584539253] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539340 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584539259.153748:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584539259.153749:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584539259.153750:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584539259.153763:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539259.153765:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539259.153768:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539259.153769:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539259.153769:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539259.153782:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584539259.153782:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584539259.153784:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21059 00000100:00080000:4.0:1584539259.153786:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584539259.153790:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539259.153792:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539259.153792:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539259.153793:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539259.153794:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539259.153795:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539259.153796:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539259.153797:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539259.194254:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584539259.194258:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584539259.194260:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584539259.194261:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584539259.194262:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584539259.194264:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584539259.194265:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584539259.194267:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539340 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584539259.194296:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078445472 new x1661492078445536 00000800:00020000:2.0:1584539265.916069:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584539265.916071:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 6, oc: 0, rc: 8 00000800:00000100:2.0:1584539265.916073:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539265.916547:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584539265.928623:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539259/real 1584539259] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539346 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584539265.928631:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584539265.928632:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584539265.928633:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584539265.928639:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539265.928640:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539265.928641:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539265.928642:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539265.928643:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539265.928645:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584539265.928645:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584539265.928647:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21065 00000100:00080000:4.0:1584539265.928649:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584539265.928657:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539265.928658:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539265.928659:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539265.928660:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539265.928660:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539265.928662:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539265.928663:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539265.928664:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539265.960384:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584539265.960388:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584539265.960389:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584539265.960390:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584539265.960393:0:25724:0:(client.c:2816:ptlrpc_resend_req()) @@@ going to resend req@ffff8d2babe45580 x1661492078445552/t0(0) o17->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 456/432 e 0 to 0 dl 1584539392 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000100:00080000:2.0:1584539265.960399:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584539265.960401:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584539265.960402:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584539265.960404:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539346 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584539265.960432:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078445536 new x1661492078445600 00000800:00000100:0.0:1584539271.141137:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539271.141825:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584539271.153912:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539265/real 1584539265] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539352 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584539271.153920:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584539271.153921:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584539271.153922:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584539271.153928:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539271.153929:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539271.153930:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539271.153931:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539271.153932:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539271.153933:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584539271.153934:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584539271.153935:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21071 00000100:00080000:4.0:1584539271.153937:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584539271.153947:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539271.153949:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539271.153950:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539271.153952:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539271.153954:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539271.153955:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539271.153956:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539271.153957:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539271.193068:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584539271.193072:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584539271.193073:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584539271.193074:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584539271.193076:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584539271.193078:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584539271.193079:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584539271.193081:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539352 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584539271.193112:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078445600 new x1661492078445648 00000800:00000100:0.0:1584539277.141387:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539277.141819:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584539277.153887:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539271/real 1584539271] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539358 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584539277.153895:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584539277.153896:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584539277.153897:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584539277.153903:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539277.153904:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539277.153905:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539277.153906:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539277.153907:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539277.153908:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584539277.153909:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584539277.153910:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21077 00000100:00080000:4.0:1584539277.153912:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584539277.153921:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539277.153922:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539277.153923:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539277.153925:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539277.153925:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539277.153926:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539277.153927:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539277.153928:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539277.183572:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584539277.183576:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584539277.183578:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584539277.183579:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584539277.183580:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584539277.183582:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584539277.183583:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584539277.183585:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539358 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584539277.183595:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078445648 new x1661492078445680 00000800:00000100:0.0:1584539283.141343:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539283.141852:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584539283.153921:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539277/real 1584539277] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539364 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584539283.153930:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584539283.153932:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584539283.153934:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584539283.153954:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539283.153956:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539283.153970:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539283.153971:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539283.153972:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539283.153974:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584539283.153974:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584539283.153977:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21083 00000100:00080000:4.0:1584539283.153990:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584539283.153999:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539283.154000:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539283.154001:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539283.154002:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539283.154003:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539283.154004:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539283.154005:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539283.154006:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539283.197082:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584539283.197088:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584539283.197091:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584539283.197093:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584539283.197095:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584539283.197097:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584539283.197099:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584539283.197101:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539364 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539283.197134:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078445680 new x1661492078445760 00000800:00020000:4.0:1584539289.916058:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584539289.916062:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584539289.916064:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539289.916544:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539289.928619:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539283/real 1584539283] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539370 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539289.928629:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539289.928630:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539289.928632:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539289.928639:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539289.928641:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539289.928642:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539289.928643:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539289.928644:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539289.928647:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539289.928647:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539289.928649:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21089 00000100:00080000:6.0:1584539289.928653:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539289.928667:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539289.928668:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539289.928669:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539289.928670:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539289.928671:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539289.928672:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539289.928673:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539289.928674:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539289.957767:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584539289.957774:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584539289.957777:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584539289.957778:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584539289.957780:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584539289.957782:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584539289.957784:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584539289.957786:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539370 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539289.957795:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078445760 new x1661492078445792 00000800:00000100:0.0:1584539295.140615:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539295.141082:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539295.153154:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539289/real 1584539289] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539376 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539295.153161:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539295.153163:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539295.153164:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539295.153169:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539295.153171:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539295.153172:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539295.153174:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539295.153174:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539295.153176:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539295.153176:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539295.153178:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21095 00000100:00080000:6.0:1584539295.153180:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539295.153191:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539295.153193:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539295.153193:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539295.153194:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539295.153195:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539295.153196:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539295.153196:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539295.153208:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539295.191215:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584539295.191220:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584539295.191222:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584539295.191223:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584539295.191225:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584539295.191227:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584539295.191228:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584539295.191230:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539376 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539295.191261:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078445792 new x1661492078445824 00000800:00000100:0.0:1584539301.140751:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539301.141155:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539301.153223:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539295/real 1584539295] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539382 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539301.153231:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539301.153233:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539301.153234:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539301.153240:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539301.153244:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539301.153245:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539301.153246:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539301.153247:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539301.153248:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539301.153249:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539301.153251:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21101 00000100:00080000:6.0:1584539301.153253:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539301.153264:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539301.153265:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539301.153266:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539301.153267:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539301.153268:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539301.153269:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539301.153269:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539301.153270:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584539301.197173:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584539301.197178:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584539301.197181:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584539301.197182:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584539301.197183:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584539301.197186:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584539301.197187:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584539301.197189:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539382 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539301.197218:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078445824 new x1661492078445872 00000800:00000100:0.0:1584539307.140908:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539307.141299:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539307.153363:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539301/real 1584539301] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539388 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539307.153370:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539307.153371:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539307.153373:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539307.153387:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539307.153388:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539307.153391:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539307.153392:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539307.153403:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539307.153404:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539307.153405:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539307.153407:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21107 00000100:00080000:6.0:1584539307.153408:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539307.153424:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539307.153425:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539307.153426:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539307.153427:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539307.153427:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539307.153429:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539307.153429:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539307.153430:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539307.192863:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584539307.192868:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584539307.192869:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584539307.192870:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584539307.192871:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584539307.192873:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584539307.192874:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584539307.192876:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539388 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539307.192909:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078445872 new x1661492078445936 00000800:00020000:4.0:1584539313.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584539313.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584539313.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539313.916525:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539313.928577:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539307/real 1584539307] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539394 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539313.928588:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539313.928590:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539313.928591:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539313.928596:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539313.928598:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539313.928598:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539313.928600:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539313.928600:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539313.928602:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539313.928602:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539313.928604:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21113 00000100:00080000:6.0:1584539313.928606:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539313.928614:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539313.928615:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539313.928616:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539313.928617:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539313.928618:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539313.928619:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539313.928620:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539313.928621:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539313.966176:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584539313.966181:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584539313.966183:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584539313.966184:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584539313.966185:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584539313.966187:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584539313.966188:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584539313.966190:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539394 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539313.966200:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078445936 new x1661492078445984 00000800:00000100:0.0:1584539319.141109:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539319.141613:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539319.153683:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539313/real 1584539313] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539400 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539319.153690:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539319.153691:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539319.153692:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539319.153697:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539319.153699:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539319.153700:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539319.153701:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539319.153701:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539319.153702:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539319.153703:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539319.153704:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21119 00000100:00080000:6.0:1584539319.153706:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539319.153713:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539319.153714:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539319.153715:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539319.153716:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539319.153716:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539319.153717:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539319.153719:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539319.153731:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539319.182600:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584539319.182604:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584539319.182605:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584539319.182606:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584539319.182607:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584539319.182609:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584539319.182611:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584539319.182612:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539400 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539319.182647:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078445984 new x1661492078446016 00000800:00000100:0.0:1584539325.141125:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539325.141553:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539325.153622:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539319/real 1584539319] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539406 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539325.153630:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539325.153631:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539325.153632:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539325.153638:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539325.153639:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539325.153640:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539325.153641:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539325.153642:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539325.153643:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539325.153644:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539325.153645:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21125 00000100:00080000:6.0:1584539325.153647:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539325.153655:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539325.153656:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539325.153657:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539325.153658:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539325.153660:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539325.153661:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539325.153661:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539325.153662:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539325.222238:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584539325.222242:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584539325.222243:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584539325.222244:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584539325.222245:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584539325.222247:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584539325.222249:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584539325.222250:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539406 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539325.222260:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078446016 new x1661492078446048 00000800:00000100:0.0:1584539331.141267:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539331.141725:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539331.153817:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539325/real 1584539325] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539412 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539331.153824:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539331.153825:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539331.153826:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539331.153840:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539331.153842:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539331.153845:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539331.153846:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539331.153847:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539331.153848:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539331.153860:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539331.153861:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21131 00000100:00080000:6.0:1584539331.153864:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539331.153869:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539331.153870:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539331.153872:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539331.153873:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539331.153874:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539331.153875:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539331.153875:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539331.153876:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539331.200808:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584539331.200812:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584539331.200814:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584539331.200815:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584539331.200816:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584539331.200818:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584539331.200819:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584539331.200821:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539412 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539331.200831:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078446048 new x1661492078446128 00000800:00000100:0.0:1584539337.141344:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539337.141830:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539337.153898:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539331/real 1584539331] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539418 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539337.153906:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539337.153907:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539337.153908:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539337.153914:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539337.153915:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539337.153916:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539337.153917:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539337.153918:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539337.153919:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539337.153920:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539337.153922:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21137 00000100:00080000:6.0:1584539337.153923:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539337.153933:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539337.153934:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539337.153934:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539337.153936:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539337.153936:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539337.153937:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539337.153938:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539337.153939:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539337.195589:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584539337.195593:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584539337.195594:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584539337.195595:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584539337.195596:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584539337.195598:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584539337.195599:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584539337.195601:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539418 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539337.195611:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078446128 new x1661492078446160 00000800:00000100:0.0:1584539343.141452:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539343.141987:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539343.154073:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539337/real 1584539337] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539424 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539343.154083:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539343.154084:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539343.154086:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539343.154095:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539343.154097:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539343.154098:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539343.154099:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539343.154100:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539343.154102:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539343.154103:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539343.154105:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21143 00000100:00080000:8.0:1584539343.154109:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539343.154123:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539343.154125:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539343.154126:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539343.154128:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539343.154128:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539343.154130:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539343.154130:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539343.154132:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539343.204727:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584539343.204736:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584539343.204738:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584539343.204739:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584539343.204741:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584539343.204744:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584539343.204745:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584539343.204747:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539424 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584539343.204776:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078446160 new x1661492078446208 00000800:00000100:0.0:1584539349.140877:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539349.141285:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584539349.153389:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539343/real 1584539343] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539430 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584539349.153398:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584539349.153400:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584539349.153402:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584539349.153409:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539349.153411:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539349.153412:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539349.153413:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539349.153414:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539349.153416:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584539349.153416:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584539349.153419:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21149 00000100:00080000:12.0:1584539349.153421:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584539349.153437:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539349.153439:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539349.153440:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539349.153441:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539349.153442:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539349.153443:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539349.153444:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539349.153445:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539349.196964:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584539349.196970:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584539349.196972:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584539349.196974:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584539349.196976:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584539349.196978:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584539349.196980:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584539349.196982:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539430 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584539349.197015:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078446208 new x1661492078446240 00000800:00000100:0.0:1584539355.140593:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539355.141021:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584539355.153075:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539349/real 1584539349] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539436 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584539355.153084:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584539355.153085:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584539355.153087:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584539355.153106:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539355.153108:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539355.153112:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539355.153113:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539355.153114:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539355.153115:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584539355.153116:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584539355.153118:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21155 00000100:00080000:12.0:1584539355.153122:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584539355.153129:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539355.153130:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539355.153131:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539355.153133:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539355.153133:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539355.153135:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539355.153135:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539355.153137:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539355.193221:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584539355.193227:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584539355.193230:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584539355.193231:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584539355.193232:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584539355.193235:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584539355.193236:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584539355.193238:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539436 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584539355.193268:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078446240 new x1661492078446304 00000800:00000100:0.0:1584539361.140723:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539361.141211:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584539361.153280:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539355/real 1584539355] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539442 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584539361.153288:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584539361.153290:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584539361.153291:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584539361.153297:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539361.153299:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539361.153299:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539361.153301:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539361.153301:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539361.153303:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584539361.153304:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584539361.153306:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21161 00000100:00080000:12.0:1584539361.153309:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584539361.153319:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539361.153320:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539361.153321:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539361.153322:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539361.153323:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539361.153324:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539361.153325:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539361.153326:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539361.194047:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584539361.194052:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584539361.194054:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584539361.194054:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584539361.194056:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584539361.194058:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584539361.194059:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584539361.194061:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539442 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584539361.194090:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078446304 new x1661492078446352 00000800:00000100:0.0:1584539367.140803:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539367.141259:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584539367.153308:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539361/real 1584539361] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539448 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584539367.153315:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584539367.153316:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584539367.153318:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584539367.153322:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539367.153324:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539367.153325:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539367.153326:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539367.153327:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539367.153329:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584539367.153329:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584539367.153331:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21167 00000100:00080000:12.0:1584539367.153333:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584539367.153341:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539367.153342:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539367.153343:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539367.153344:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539367.153344:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539367.153346:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539367.153346:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539367.153347:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539367.362788:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584539367.362792:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584539367.362793:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584539367.362794:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584539367.362796:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584539367.362797:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584539367.362799:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584539367.362801:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539448 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584539367.362830:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078446352 new x1661492078446384 00000800:00000100:0.0:1584539373.140853:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539373.141284:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584539373.153334:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539367/real 1584539367] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539454 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584539373.153342:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584539373.153343:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584539373.153344:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584539373.153350:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539373.153351:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539373.153353:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539373.153354:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539373.153355:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539373.153356:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584539373.153357:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584539373.153358:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21173 00000100:00080000:12.0:1584539373.153360:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584539373.153368:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539373.153370:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539373.153370:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539373.153371:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539373.153372:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539373.153373:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539373.153374:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539373.153375:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539373.186995:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584539373.186999:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584539373.187000:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584539373.187001:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584539373.187002:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584539373.187004:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584539373.187006:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584539373.187007:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539454 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584539373.187036:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078446384 new x1661492078446432 00000800:00000100:0.0:1584539379.141164:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539379.141565:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584539379.153615:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539373/real 1584539373] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539460 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584539379.153622:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584539379.153623:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584539379.153625:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584539379.153640:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539379.153641:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539379.153650:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539379.153651:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539379.153652:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539379.153653:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584539379.153654:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584539379.153656:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21179 00000100:00080000:12.0:1584539379.153658:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584539379.153663:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539379.153664:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539379.153665:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539379.153666:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539379.153666:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539379.153668:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539379.153668:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539379.153669:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539379.161798:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584539379.161802:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584539379.161803:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584539379.161804:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584539379.161805:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584539379.161807:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584539379.161809:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584539379.161810:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539460 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584539379.161839:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078446432 new x1661492078446496 00000800:00000100:0.0:1584539385.141596:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539385.142060:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584539385.154111:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539379/real 1584539379] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539466 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584539385.154119:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584539385.154121:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584539385.154122:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584539385.154128:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539385.154129:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539385.154130:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539385.154131:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539385.154132:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539385.154133:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584539385.154134:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584539385.154135:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21185 00000100:00080000:12.0:1584539385.154137:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584539385.154146:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539385.154147:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539385.154147:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539385.154149:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539385.154149:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539385.154150:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539385.154151:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539385.154152:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539385.164190:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584539385.164194:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584539385.164195:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584539385.164196:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584539385.164197:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584539385.164199:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584539385.164200:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584539385.164202:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539466 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584539385.164231:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078446496 new x1661492078446528 00000800:00000100:0.0:1584539391.141192:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539391.141689:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584539391.153744:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539385/real 1584539385] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539472 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584539391.153753:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584539391.153754:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584539391.153756:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584539391.153762:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539391.153764:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539391.153764:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539391.153765:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539391.153766:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539391.153767:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584539391.153768:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584539391.153770:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21191 00000100:00080000:12.0:1584539391.153772:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584539391.153780:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539391.153781:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539391.153782:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539391.153783:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539391.153784:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539391.153785:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539391.153785:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539391.153787:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539391.161246:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584539391.161250:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584539391.161251:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584539391.161252:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584539391.161253:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584539391.161255:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584539391.161257:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584539391.161258:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539472 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584539391.161287:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078446528 new x1661492078446576 00000100:00000400:10.0:1584539392.825169:0:25726:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584539260/real 1584539260] req@ffff8d2babe47080 x1661492078445568/t0(0) o17->public1-OST0001-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 456/432 e 0 to 1 dl 1584539392 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 00000100:02000400:10.0:1584539392.825177:0:25726:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection to public1-OST0001 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539392.825179:0:25726:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539392.825181:0:25726:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0001_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539392.825186:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539392.825188:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539392.825189:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539392.825190:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:10.0:1584539392.825191:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539392.825192:0:334:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 20003 00000100:00080000:10.0:1584539392.825195:0:334:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539392.825204:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539392.825205:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539392.825206:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539392.825207:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539392.825208:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539392.825209:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539392.825209:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539392.825210:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539392.825211:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539392.825212:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000800:00000100:0.0:1584539397.141492:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539397.141894:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584539397.153944:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539391/real 1584539391] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539478 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584539397.153951:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584539397.153952:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584539397.153954:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584539397.153959:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539397.153961:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539397.153962:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539397.153963:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539397.153964:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539397.153965:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584539397.153966:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584539397.153967:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21197 00000100:00080000:12.0:1584539397.153969:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584539397.153978:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539397.153979:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539397.153980:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539397.153981:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539397.153981:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539397.153982:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539397.153984:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539397.153985:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539397.161789:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584539397.161793:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584539397.161794:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584539397.161795:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584539397.161796:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584539397.161798:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584539397.161800:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584539397.161801:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539478 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584539397.161830:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078446576 new x1661492078446624 00000100:00080000:8.0:1584539398.153078:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:8.0:1584539398.153080:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0001_UUID on 10.10.2.22@o2ib failed (-110) 00000800:00000100:0.0:1584539403.141392:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539403.141823:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584539403.153874:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539397/real 1584539397] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539484 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584539403.153882:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584539403.153883:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584539403.153884:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584539403.153898:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539403.153899:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539403.153900:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539403.153902:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539403.153902:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539403.153904:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584539403.153904:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584539403.153906:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21203 00000100:00080000:12.0:1584539403.153909:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584539403.153914:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539403.153915:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539403.153916:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539403.153917:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539403.153919:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539403.153920:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539403.153921:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539403.153922:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539403.161811:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584539403.161817:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584539403.161819:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584539403.161820:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584539403.161821:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584539403.161823:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584539403.161825:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584539403.161827:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539484 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584539403.161856:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078446624 new x1661492078446688 00000800:00000100:0.0:1584539409.141560:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539409.142001:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584539409.154050:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539403/real 1584539403] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539490 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584539409.154058:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584539409.154059:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584539409.154061:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584539409.154066:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539409.154068:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539409.154068:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539409.154070:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539409.154070:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539409.154072:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584539409.154072:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584539409.154074:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21209 00000100:00080000:12.0:1584539409.154076:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584539409.154086:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539409.154087:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539409.154089:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539409.154090:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539409.154090:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539409.154092:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539409.154092:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539409.154093:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539409.162086:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584539409.162091:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584539409.162093:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584539409.162094:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584539409.162095:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584539409.162097:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584539409.162098:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584539409.162100:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539490 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584539409.162129:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078446688 new x1661492078446720 00000800:00000100:0.0:1584539415.141511:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539415.141943:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584539415.154017:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539409/real 1584539409] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539496 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584539415.154026:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584539415.154042:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584539415.154043:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584539415.154050:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539415.154051:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539415.154052:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:12.0:1584539415.154054:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:12.0:1584539415.154055:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584539415.154057:0:4059:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21198 00000100:00080000:12.0:1584539415.154058:0:4059:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 19979 00000100:00080000:12.0:1584539415.154061:0:4059:0:(import.c:596:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0001 (at 10.10.2.21@o2ib) 00000100:00080000:12.0:1584539415.154062:0:4059:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.21@o2ib/10.10.2.21@o2ib 00000100:00080000:12.0:1584539415.154076:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539415.154077:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539415.154078:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539415.154079:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584539415.154080:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584539415.154081:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21215 00000100:00080000:12.0:1584539415.154082:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584539415.154085:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539415.154087:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539415.154087:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539415.154089:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539415.154089:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539415.154090:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539415.154091:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539415.154092:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539415.154286:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:8.0:1584539415.154288:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0001_UUID on 10.10.2.21@o2ib failed (-19) 00000100:00080000:8.0:1584539415.178874:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584539415.178879:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584539415.178881:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584539415.178882:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584539415.178884:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584539415.178886:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584539415.178888:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584539415.178889:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539496 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584539415.178918:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078446720 new x1661492078446768 00000800:00000100:0.0:1584539421.140165:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539421.140616:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584539421.152663:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539415/real 1584539415] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539502 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584539421.152674:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584539421.152675:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584539421.152677:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584539421.152685:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539421.152687:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539421.152688:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539421.152690:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539421.152690:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539421.152692:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584539421.152693:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584539421.152695:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21221 00000100:00080000:12.0:1584539421.152698:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584539421.152711:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539421.152712:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539421.152713:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539421.152714:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539421.152715:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539421.152716:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539421.152717:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539421.152718:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539421.202492:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584539421.202499:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584539421.202501:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584539421.202502:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584539421.202505:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584539421.202507:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584539421.202509:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584539421.202511:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539502 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584539421.202520:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078446768 new x1661492078446816 00000800:00000100:0.0:1584539427.137118:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539427.137611:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584539427.149655:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539421/real 1584539421] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539508 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584539427.149663:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584539427.149665:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584539427.149666:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584539427.149680:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539427.149682:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539427.149683:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539427.149684:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539427.149684:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539427.149686:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584539427.149686:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584539427.149688:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21227 00000100:00080000:12.0:1584539427.149690:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584539427.149695:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539427.149697:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539427.149697:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539427.149699:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539427.149699:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539427.149700:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539427.149701:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539427.149702:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539427.195412:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584539427.195416:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584539427.195417:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584539427.195418:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584539427.195419:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584539427.195421:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584539427.195423:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584539427.195425:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539508 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584539427.195454:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078446816 new x1661492078446864 00000800:00000100:0.0:1584539433.134262:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539433.134682:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584539433.146728:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539427/real 1584539427] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539514 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584539433.146737:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584539433.146738:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584539433.146740:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584539433.146745:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539433.146747:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539433.146748:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539433.146749:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539433.146749:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539433.146751:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584539433.146751:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584539433.146753:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21233 00000100:00080000:12.0:1584539433.146755:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584539433.146765:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539433.146766:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539433.146767:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539433.146768:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539433.146768:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539433.146770:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539433.146770:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539433.146771:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539433.193780:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584539433.193785:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584539433.193786:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584539433.193787:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584539433.193789:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584539433.193790:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584539433.193792:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584539433.193794:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539514 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584539433.193822:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078446864 new x1661492078446912 00000800:00020000:2.0:1584539439.916050:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584539439.916052:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584539439.916054:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539439.916499:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584539439.928544:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539433/real 1584539433] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539520 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584539439.928552:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584539439.928553:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584539439.928554:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584539439.928559:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539439.928561:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539439.928561:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:12.0:1584539439.928563:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:12.0:1584539439.928564:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584539439.928565:0:4059:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21198 00000100:00080000:12.0:1584539439.928567:0:4059:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 21221 00000100:00080000:12.0:1584539439.928568:0:4059:0:(import.c:570:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: tried all connections, increasing latency to 6s 00000100:00080000:12.0:1584539439.928570:0:4059:0:(import.c:596:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0001 (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584539439.928571:0:4059:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584539439.928579:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539439.928581:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539439.928581:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539439.928582:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584539439.928583:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584539439.928584:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21239 00000100:00080000:12.0:1584539439.928586:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584539439.928590:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539439.928591:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539439.928592:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539439.928593:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539439.928593:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539439.928594:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539439.928595:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539439.928596:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539439.979747:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584539439.979751:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584539439.979752:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584539439.979753:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584539439.979755:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584539439.979757:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584539439.979759:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584539439.979760:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539520 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584539439.979768:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078446912 new x1661492078446960 00000100:00080000:8.0:1584539439.979768:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584539439.979770:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584539439.979770:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0001_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584539439.979771:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584539439.979772:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584539439.979773:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584539439.979774:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584539439.979775:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727f00): req@ffff8d2babe47080 x1661492078445568/t0(0) o17->public1-OST0001-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 456/432 e 0 to 1 dl 1584539392 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 00000800:00000100:0.0:1584539445.131018:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539445.131442:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584539445.143492:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539439/real 1584539439] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539526 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584539445.143500:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584539445.143501:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584539445.143502:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584539445.143507:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539445.143508:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539445.143509:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539445.143510:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539445.143511:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539445.143512:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584539445.143514:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584539445.143515:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21245 00000100:00080000:12.0:1584539445.143517:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584539445.143525:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539445.143526:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539445.143527:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539445.143528:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539445.143528:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539445.143530:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539445.143530:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539445.143531:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539445.193041:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584539445.193045:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584539445.193047:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584539445.193047:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584539445.193049:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584539445.193050:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584539445.193052:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584539445.193054:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539526 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584539445.193062:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078446960 new x1661492078446992 00000800:00000100:0.0:1584539451.130984:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539451.131406:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584539451.143478:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539445/real 1584539445] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539532 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584539451.143486:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584539451.143487:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584539451.143489:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584539451.143502:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539451.143504:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539451.143505:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539451.143506:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539451.143507:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539451.143509:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584539451.143509:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584539451.143511:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21251 00000100:00080000:12.0:1584539451.143513:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584539451.143518:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539451.143519:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539451.143520:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539451.143521:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539451.143522:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539451.143523:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539451.143523:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584539451.143525:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539451.203739:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584539451.203743:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584539451.203744:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584539451.203745:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584539451.203748:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584539451.203749:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584539451.203751:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584539451.203753:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539532 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584539451.203782:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078446992 new x1661492078447056 00000800:00000100:0.0:1584539457.131197:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539457.131611:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584539457.143660:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539451/real 1584539451] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539538 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584539457.143668:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584539457.143669:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584539457.143671:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584539457.143677:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584539457.143679:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539457.143680:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584539457.143681:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539457.143683:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584539457.143684:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584539457.143685:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584539457.143686:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21257 00000100:00080000:14.0:1584539457.143688:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584539457.143698:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584539457.143699:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539457.143700:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584539457.143701:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539457.143702:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584539457.143703:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539457.143704:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584539457.143705:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539457.190495:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584539457.190498:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584539457.190500:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584539457.190501:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584539457.190502:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584539457.190504:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584539457.190505:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584539457.190507:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539538 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584539457.190537:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078447056 new x1661492078447088 00000800:00020000:2.0:1584539463.916121:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584539463.916125:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584539463.916127:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539463.916512:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584539463.928640:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539457/real 1584539457] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539544 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584539463.928650:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584539463.928652:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584539463.928654:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584539463.928662:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584539463.928665:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539463.928681:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584539463.928682:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539463.928683:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584539463.928685:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584539463.928686:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584539463.928688:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21263 00000100:00080000:14.0:1584539463.928691:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584539463.928699:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584539463.928700:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539463.928701:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584539463.928702:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539463.928703:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584539463.928705:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539463.928705:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584539463.928706:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539463.967593:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584539463.967600:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584539463.967603:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584539463.967604:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584539463.967606:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584539463.967608:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584539463.967610:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584539463.967612:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539544 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584539463.967621:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078447088 new x1661492078447152 00000800:00000100:0.0:1584539469.131954:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539469.132365:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584539469.144421:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539463/real 1584539463] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539550 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584539469.144430:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584539469.144432:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584539469.144433:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584539469.144439:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539469.144441:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539469.144442:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539469.144444:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539469.144444:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539469.144446:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584539469.144447:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584539469.144449:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21269 00000100:00080000:18.0:1584539469.144451:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584539469.144466:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539469.144467:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539469.144468:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539469.144470:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539469.144471:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539469.144472:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539469.144473:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539469.144474:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539469.201334:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584539469.201340:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584539469.201341:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584539469.201343:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584539469.201344:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584539469.201347:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584539469.201348:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584539469.201350:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539550 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584539469.201361:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078447152 new x1661492078447184 00000800:00000100:0.0:1584539475.134908:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539475.135300:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584539475.147380:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539469/real 1584539469] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539556 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584539475.147389:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584539475.147390:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584539475.147392:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584539475.147412:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539475.147414:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539475.147414:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539475.147416:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539475.147416:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539475.147418:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584539475.147419:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584539475.147421:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21275 00000100:00080000:18.0:1584539475.147423:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584539475.147430:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539475.147433:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539475.147433:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539475.147435:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539475.147435:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539475.147437:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539475.147437:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539475.147438:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539475.202006:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584539475.202012:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584539475.202014:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584539475.202015:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584539475.202016:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584539475.202018:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584539475.202020:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584539475.202022:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539556 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584539475.202061:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078447184 new x1661492078447232 00000800:00000100:0.0:1584539481.138238:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539481.138656:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584539481.150716:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539475/real 1584539475] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539562 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584539481.150724:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584539481.150726:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584539481.150727:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584539481.150732:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539481.150734:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539481.150735:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539481.150736:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539481.150737:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539481.150738:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584539481.150739:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584539481.150741:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21281 00000100:00080000:18.0:1584539481.150744:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584539481.150753:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539481.150755:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539481.150756:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539481.150757:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539481.150758:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539481.150759:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539481.150759:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539481.150761:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539481.194724:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584539481.194729:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584539481.194731:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584539481.194732:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584539481.194733:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584539481.194735:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584539481.194737:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584539481.194739:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539562 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584539481.194769:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078447232 new x1661492078447280 00000800:00020000:2.0:1584539487.916430:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584539487.916433:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584539487.916435:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539487.916872:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584539487.928929:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539481/real 1584539481] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539568 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584539487.928937:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584539487.928938:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584539487.928940:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584539487.928945:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539487.928946:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539487.928956:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539487.928957:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539487.928958:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539487.928959:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584539487.928961:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584539487.928962:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21287 00000100:00080000:18.0:1584539487.928964:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584539487.928970:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539487.928971:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539487.928972:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539487.928973:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539487.928974:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539487.928975:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539487.928975:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539487.928976:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539487.979080:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584539487.979084:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584539487.979086:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584539487.979087:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584539487.979088:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584539487.979090:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584539487.979092:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584539487.979094:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539568 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584539487.979123:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078447280 new x1661492078447328 00000800:00000100:0.0:1584539493.144307:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539493.144739:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584539493.156798:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539487/real 1584539487] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539574 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584539493.156806:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584539493.156807:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584539493.156809:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584539493.156815:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539493.156816:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539493.156817:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539493.156818:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539493.156820:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539493.156821:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584539493.156822:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584539493.156824:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21293 00000100:00080000:18.0:1584539493.156826:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584539493.156834:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539493.156835:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539493.156836:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539493.156837:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539493.156838:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539493.156839:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539493.156840:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539493.156841:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539493.197579:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584539493.197583:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584539493.197584:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584539493.197585:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584539493.197587:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584539493.197588:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584539493.197590:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584539493.197592:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539574 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584539493.197600:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078447328 new x1661492078447376 00000800:00000100:0.0:1584539499.145947:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539499.146382:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584539499.158454:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539493/real 1584539493] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539580 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584539499.158462:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584539499.158463:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584539499.158464:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584539499.158479:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539499.158480:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539499.158482:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539499.158483:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539499.158484:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539499.158485:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584539499.158486:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584539499.158487:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21299 00000100:00080000:18.0:1584539499.158489:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584539499.158495:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539499.158496:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539499.158497:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539499.158498:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539499.158498:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539499.158499:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539499.158500:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539499.158501:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539499.205961:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584539499.205965:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584539499.205967:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584539499.205968:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584539499.205969:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584539499.205971:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584539499.205973:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584539499.205974:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539580 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584539499.206008:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078447376 new x1661492078447424 00000800:00000100:0.0:1584539505.146215:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539505.146717:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584539505.158799:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539499/real 1584539499] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539586 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584539505.158809:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584539505.158810:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584539505.158812:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584539505.158821:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539505.158823:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539505.158824:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539505.158825:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539505.158826:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539505.158828:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584539505.158828:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584539505.158830:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21305 00000100:00080000:18.0:1584539505.158833:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584539505.158848:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539505.158849:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539505.158850:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539505.158851:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539505.158852:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539505.158853:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539505.158854:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539505.158855:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539505.203060:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584539505.203066:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584539505.203068:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584539505.203069:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584539505.203071:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584539505.203073:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584539505.203075:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584539505.203077:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539586 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584539505.203107:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078447424 new x1661492078447456 00000800:00020000:2.0:1584539511.916082:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584539511.916085:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584539511.916087:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539511.916532:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584539511.928588:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539505/real 1584539505] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539592 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584539511.928596:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584539511.928598:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584539511.928600:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584539511.928607:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539511.928609:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539511.928620:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539511.928621:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539511.928622:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539511.928623:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584539511.928624:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584539511.928626:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21311 00000100:00080000:18.0:1584539511.928628:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584539511.928635:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539511.928636:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539511.928637:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539511.928638:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539511.928639:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539511.928640:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539511.928641:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539511.928642:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539512.516678:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584539512.516684:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584539512.516686:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584539512.516687:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584539512.516689:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584539512.516691:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584539512.516692:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584539512.516694:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539592 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584539512.516706:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078447456 new x1661492078447520 00000800:00000100:2.0:1584539518.146391:0:4532:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539518.146840:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584539518.158916:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539512/real 1584539512] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539599 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584539518.158925:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584539518.158926:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584539518.158928:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584539518.158933:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539518.158935:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539518.158936:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539518.158937:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539518.158938:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539518.158939:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584539518.158940:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584539518.158942:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21317 00000100:00080000:18.0:1584539518.158944:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584539518.158952:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539518.158953:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539518.158954:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539518.158955:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539518.158956:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539518.158957:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539518.158958:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539518.158959:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539518.205799:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584539518.205803:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584539518.205806:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584539518.205807:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584539518.205808:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584539518.205810:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584539518.205811:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584539518.205813:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539599 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584539518.205842:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078447520 new x1661492078447552 00000800:00000100:0.0:1584539524.146459:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539524.146897:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584539524.158968:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539518/real 1584539518] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539605 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584539524.158978:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584539524.158980:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584539524.158982:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584539524.159001:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539524.159003:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539524.159004:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539524.159005:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539524.159006:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539524.159008:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584539524.159009:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584539524.159011:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21324 00000100:00080000:18.0:1584539524.159014:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584539524.159022:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539524.159024:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539524.159025:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539524.159026:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539524.159042:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539524.159043:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539524.159045:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539524.159046:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539524.191222:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584539524.191228:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584539524.191231:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584539524.191232:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584539524.191234:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584539524.191236:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584539524.191238:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584539524.191240:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539605 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584539524.191270:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078447552 new x1661492078447616 00000800:00020000:2.0:1584539530.916039:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584539530.916042:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584539530.916044:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539530.916509:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584539530.928594:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539524/real 1584539524] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539611 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584539530.928614:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584539530.928616:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584539530.928618:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584539530.928626:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539530.928628:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539530.928629:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539530.928630:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539530.928631:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539530.928633:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584539530.928634:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584539530.928636:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21330 00000100:00080000:18.0:1584539530.928638:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584539530.928654:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539530.928655:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539530.928656:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539530.928657:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539530.928659:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539530.928660:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539530.928661:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539530.928662:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539530.965922:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584539530.965929:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584539530.965931:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584539530.965933:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584539530.965934:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584539530.965937:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584539530.965938:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584539530.965940:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539611 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584539530.965974:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078447616 new x1661492078447648 00000800:00000100:0.0:1584539536.140186:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539536.140655:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584539536.152719:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539530/real 1584539530] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539617 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584539536.152727:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584539536.152728:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584539536.152730:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584539536.152736:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539536.152738:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539536.152749:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539536.152751:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539536.152751:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539536.152753:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584539536.152754:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584539536.152755:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21336 00000100:00080000:18.0:1584539536.152768:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584539536.152774:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539536.152776:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539536.152778:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539536.152779:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539536.152779:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539536.152781:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539536.152781:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539536.152782:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539536.206259:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584539536.206265:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584539536.206266:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584539536.206267:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584539536.206269:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584539536.206271:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584539536.206273:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584539536.206275:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539617 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584539536.206305:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078447648 new x1661492078447696 00000800:00000100:0.0:1584539542.135692:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539542.136121:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584539542.148175:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539536/real 1584539536] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539623 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584539542.148184:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584539542.148185:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584539542.148186:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584539542.148192:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539542.148194:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539542.148195:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539542.148196:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539542.148197:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539542.148199:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584539542.148199:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584539542.148201:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21342 00000100:00080000:18.0:1584539542.148203:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584539542.148214:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539542.148215:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539542.148216:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539542.148217:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539542.148218:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539542.148219:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539542.148220:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539542.148221:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539542.181296:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584539542.181302:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584539542.181304:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584539542.181305:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584539542.181306:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584539542.181308:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584539542.181310:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584539542.181312:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539623 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584539542.181341:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078447696 new x1661492078447744 00000800:00000100:0.0:1584539548.134539:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539548.134976:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584539548.147024:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539542/real 1584539542] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539629 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584539548.147081:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584539548.147082:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584539548.147084:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584539548.147098:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539548.147100:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539548.147101:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539548.147102:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539548.147102:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539548.147104:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584539548.147104:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584539548.147107:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21348 00000100:00080000:18.0:1584539548.147109:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584539548.147115:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539548.147116:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539548.147116:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539548.147118:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539548.147118:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539548.147119:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539548.147120:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539548.147121:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539548.195300:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584539548.195305:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584539548.195306:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584539548.195307:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584539548.195308:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584539548.195310:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584539548.195311:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584539548.195313:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539629 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584539548.195348:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078447744 new x1661492078447792 00000800:00020000:2.0:1584539554.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584539554.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584539554.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539554.916550:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584539554.928636:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539548/real 1584539548] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539635 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584539554.928655:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584539554.928656:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584539554.928657:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584539554.928663:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539554.928665:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539554.928665:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539554.928667:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539554.928667:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539554.928669:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584539554.928670:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584539554.928672:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21354 00000100:00080000:18.0:1584539554.928673:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584539554.928682:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539554.928683:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539554.928684:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539554.928685:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539554.928686:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539554.928687:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539554.928687:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539554.928688:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539554.976008:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584539554.976012:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584539554.976025:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584539554.976025:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584539554.976058:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584539554.976060:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584539554.976061:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584539554.976063:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539635 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584539554.976095:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078447792 new x1661492078447840 00000800:00000100:0.0:1584539560.133546:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539560.134063:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584539560.146118:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539554/real 1584539554] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539641 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584539560.146126:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584539560.146127:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584539560.146128:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584539560.146134:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539560.146135:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539560.146144:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539560.146146:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539560.146147:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539560.146148:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584539560.146149:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584539560.146150:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21360 00000100:00080000:18.0:1584539560.146152:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584539560.146158:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539560.146159:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539560.146160:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539560.146161:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539560.146161:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539560.146163:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539560.146163:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539560.146164:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539560.182734:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584539560.182738:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584539560.182739:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584539560.182740:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584539560.182741:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584539560.182743:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584539560.182745:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584539560.182746:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539641 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584539560.182774:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078447840 new x1661492078447888 00000800:00000100:0.0:1584539566.133825:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539566.134292:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584539566.146342:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539560/real 1584539560] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539647 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584539566.146350:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584539566.146351:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584539566.146353:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584539566.146359:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539566.146361:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539566.146362:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539566.146363:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539566.146364:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539566.146365:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584539566.146366:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584539566.146367:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21366 00000100:00080000:18.0:1584539566.146369:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584539566.146377:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539566.146379:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539566.146379:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539566.146381:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539566.146381:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539566.146382:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539566.146383:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539566.146384:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539566.282151:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584539566.282155:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584539566.282156:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584539566.282157:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584539566.282158:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584539566.282160:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584539566.282162:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584539566.282163:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539647 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584539566.282192:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078447888 new x1661492078447920 00000800:00000100:0.0:1584539572.134097:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539572.134569:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584539572.146621:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539566/real 1584539566] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539653 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584539572.146629:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584539572.146630:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584539572.146632:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584539572.146647:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539572.146649:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539572.146649:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539572.146650:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539572.146651:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539572.146652:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584539572.146653:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584539572.146655:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21372 00000100:00080000:18.0:1584539572.146657:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584539572.146662:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539572.146664:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539572.146664:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539572.146665:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539572.146666:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539572.146667:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539572.146668:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539572.146669:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539572.188300:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584539572.188304:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584539572.188306:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584539572.188306:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584539572.188308:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584539572.188309:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584539572.188311:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584539572.188313:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539653 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584539572.188346:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078447920 new x1661492078447984 00000800:00020000:4.0:1584539578.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584539578.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584539578.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539578.916490:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584539578.928541:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539572/real 1584539572] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539659 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584539578.928554:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584539578.928555:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584539578.928556:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584539578.928562:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539578.928563:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539578.928564:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539578.928565:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539578.928566:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539578.928567:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584539578.928568:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584539578.928569:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21378 00000100:00080000:18.0:1584539578.928571:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584539578.928579:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539578.928581:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539578.928581:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539578.928582:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539578.928583:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539578.928584:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539578.928585:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539578.928586:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539578.982381:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584539578.982385:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584539578.982386:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584539578.982387:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584539578.982388:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584539578.982390:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584539578.982391:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584539578.982393:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539659 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584539578.982423:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078447984 new x1661492078448016 00000800:00000100:0.0:1584539584.134114:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539584.134575:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584539584.146628:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539578/real 1584539578] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539665 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584539584.146636:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584539584.146637:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584539584.146639:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584539584.146645:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539584.146647:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539584.146656:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539584.146657:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539584.146658:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539584.146659:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584539584.146660:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584539584.146662:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21384 00000100:00080000:18.0:1584539584.146664:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584539584.146670:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539584.146671:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539584.146672:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539584.146673:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539584.146673:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539584.146675:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539584.146676:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539584.146677:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539584.202545:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584539584.202551:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584539584.202552:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584539584.202553:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584539584.202555:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584539584.202556:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584539584.202558:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584539584.202560:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539665 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584539584.202588:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078448016 new x1661492078448080 00000800:00000100:0.0:1584539590.134132:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539590.134575:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584539590.146651:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539584/real 1584539584] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539671 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584539590.146660:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584539590.146661:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584539590.146663:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584539590.146670:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539590.146672:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539590.146673:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539590.146674:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539590.146675:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539590.146677:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584539590.146677:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584539590.146679:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21390 00000100:00080000:18.0:1584539590.146681:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584539590.146694:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539590.146695:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539590.146696:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539590.146697:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539590.146698:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539590.146700:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539590.146700:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539590.146713:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539590.196427:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584539590.196433:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584539590.196434:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584539590.196435:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584539590.196437:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584539590.196439:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584539590.196440:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584539590.196442:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539671 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584539590.196472:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078448080 new x1661492078448112 00000800:00000100:0.0:1584539596.136626:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539596.137069:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584539596.149138:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539590/real 1584539590] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539677 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584539596.149149:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584539596.149151:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584539596.149153:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584539596.149176:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539596.149178:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539596.149179:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539596.149181:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539596.149181:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539596.149183:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584539596.149184:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584539596.149187:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21396 00000100:00080000:18.0:1584539596.149190:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584539596.149198:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539596.149200:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539596.149202:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539596.149203:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539596.149204:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539596.149205:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539596.149206:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584539596.149207:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539596.194508:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584539596.194515:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584539596.194517:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584539596.194518:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584539596.194520:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584539596.194523:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584539596.194524:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584539596.194526:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539677 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584539596.194555:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078448112 new x1661492078448160 00000800:00020000:4.0:1584539602.916154:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584539602.916158:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584539602.916160:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539602.916668:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584539602.928722:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539596/real 1584539596] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539683 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584539602.972647:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584539602.972659:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584539602.972661:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584539602.972667:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539602.972669:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539602.972670:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539602.972671:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539602.972672:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539602.972674:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584539602.972674:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584539602.972676:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21402 00000100:00080000:20.0:1584539602.972679:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584539602.972694:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539602.972695:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539602.972696:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539602.972698:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539602.972699:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539602.972700:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539602.972701:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539602.972702:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539603.025984:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584539603.025991:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584539603.025993:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584539603.025995:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584539603.025997:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584539603.025999:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584539603.026000:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584539603.026002:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539683 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584539603.026033:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078448160 new x1661492078448208 00000800:00000100:0.0:1584539608.144587:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539608.144990:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584539608.157049:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539603/real 1584539603] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539690 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584539608.157057:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584539608.157058:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584539608.157059:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584539608.157065:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539608.157066:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539608.157075:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539608.157076:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539608.157077:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539608.157078:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584539608.157079:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584539608.157080:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21408 00000100:00080000:20.0:1584539608.157082:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584539608.157088:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539608.157091:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539608.157091:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539608.157092:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539608.157093:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539608.157094:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539608.157095:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539608.157096:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539608.202819:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584539608.202823:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584539608.202824:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584539608.202825:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584539608.202826:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584539608.202828:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584539608.202829:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584539608.202831:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539690 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584539608.202860:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078448208 new x1661492078448256 00000800:00000100:0.0:1584539614.148277:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539614.148708:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584539614.160761:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539608/real 1584539608] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539695 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584539614.160769:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584539614.160771:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584539614.160772:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584539614.160777:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539614.160779:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539614.160780:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539614.160781:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539614.160781:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539614.160783:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584539614.160783:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584539614.160785:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21414 00000100:00080000:20.0:1584539614.160788:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584539614.160797:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539614.160798:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539614.160799:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539614.160800:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539614.160801:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539614.160802:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539614.160802:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539614.160803:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584539614.208851:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584539614.208855:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584539614.208857:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584539614.208857:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584539614.208859:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584539614.208861:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584539614.208862:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584539614.208864:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539695 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584539614.208893:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078448256 new x1661492078448304 00000800:00000100:0.0:1584539620.149956:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539620.150391:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584539620.162445:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539614/real 1584539614] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539701 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584539620.162453:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584539620.162454:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584539620.162455:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584539620.162468:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539620.162470:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539620.162471:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539620.162472:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539620.162472:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539620.162474:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584539620.162475:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584539620.162477:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21420 00000100:00080000:20.0:1584539620.162479:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584539620.162485:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539620.162486:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539620.162487:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539620.162488:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539620.162489:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539620.162490:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539620.162490:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539620.162492:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539620.210923:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539620.210927:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539620.210928:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539620.210929:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539620.210930:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539620.210932:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539620.210934:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539620.210937:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539701 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584539620.210967:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078448304 new x1661492078448352 00000800:00020000:2.0:1584539626.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584539626.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584539626.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539626.916491:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584539626.928572:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539620/real 1584539620] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539707 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584539626.928579:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584539626.928581:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584539626.928582:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584539626.928586:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539626.928588:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539626.928589:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539626.928590:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539626.928591:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539626.928593:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584539626.928594:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584539626.928595:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21426 00000100:00080000:20.0:1584539626.928597:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584539626.928605:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539626.928607:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539626.928607:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539626.928609:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539626.928609:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539626.928610:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539626.928611:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539626.928612:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539626.964590:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539626.964594:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539626.964595:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539626.964596:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539626.964597:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539626.964599:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539626.964600:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539626.964602:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539707 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584539626.964635:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078448352 new x1661492078448400 00000800:00000100:0.0:1584539632.149670:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539632.150091:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584539632.162143:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539626/real 1584539626] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539713 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584539632.162151:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584539632.187612:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584539632.187613:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584539632.187618:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539632.187620:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539632.187630:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539632.187631:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539632.187632:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539632.187633:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584539632.187634:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584539632.187636:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21432 00000100:00080000:20.0:1584539632.187638:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584539632.187644:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539632.187645:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539632.187646:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539632.187647:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539632.187648:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539632.187649:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584539632.187649:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584539632.187650:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539632.229111:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539632.229115:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539632.229116:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539632.229117:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539632.229118:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539632.229120:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539632.229122:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539632.247019:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539713 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584539632.247057:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078448400 new x1661492078448448 00000800:00000100:0.0:1584539638.149811:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539638.150220:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584539638.162270:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539632/real 1584539632] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539719 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584539638.162278:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584539638.162279:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584539638.162281:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584539638.162286:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539638.162288:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539638.162289:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539638.162290:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539638.162291:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539638.162292:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584539638.162293:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584539638.162294:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21438 00000100:00080000:22.0:1584539638.162296:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584539638.162307:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539638.162308:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539638.162308:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539638.162310:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539638.162310:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539638.162311:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539638.162312:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539638.162314:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539638.203050:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539638.203054:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539638.203055:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539638.203056:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539638.203057:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539638.203059:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539638.203061:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539638.203063:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539719 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584539638.203072:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078448448 new x1661492078448480 00000800:00000100:0.0:1584539644.150024:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539644.150505:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584539644.162559:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539638/real 1584539638] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539725 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584539644.162569:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584539644.162571:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584539644.162573:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584539644.162592:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539644.162594:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539644.162596:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539644.162597:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539644.162598:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539644.162599:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584539644.162600:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584539644.162603:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21444 00000100:00080000:22.0:1584539644.162606:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584539644.162614:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539644.162616:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539644.162616:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539644.162618:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539644.162618:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539644.162621:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539644.162622:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539644.162623:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539644.210903:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539644.210909:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539644.210911:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539644.210912:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539644.210914:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539644.210917:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539644.210918:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539644.210920:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539725 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584539644.210928:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078448480 new x1661492078448544 00000800:00020000:4.0:1584539650.916036:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584539650.916040:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584539650.916042:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539650.916495:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584539650.928547:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539644/real 1584539644] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539731 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584539650.928559:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584539650.928560:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584539650.928562:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584539650.928572:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539650.928574:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539650.928575:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539650.928576:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539650.928577:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539650.928578:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584539650.928579:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584539650.928582:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21450 00000100:00080000:22.0:1584539650.928585:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584539650.928602:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539650.928604:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539650.928604:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539650.928607:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539650.928608:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539650.928609:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539650.928610:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539650.928611:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539650.962540:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539650.962548:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539650.962550:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539650.962552:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539650.962553:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539650.962556:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539650.962558:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539650.962560:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539731 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584539650.962569:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078448544 new x1661492078448576 00000800:00000100:0.0:1584539656.145959:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539656.146348:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584539656.158391:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539650/real 1584539650] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539737 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584539656.158399:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584539656.158400:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584539656.158401:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584539656.158407:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539656.158409:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539656.158418:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539656.158419:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539656.158420:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539656.158421:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584539656.158422:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584539656.158424:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21456 00000100:00080000:22.0:1584539656.158425:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584539656.158431:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539656.158433:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539656.158433:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539656.158435:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539656.158435:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539656.158436:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539656.158437:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539656.158438:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539656.216906:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539656.216910:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539656.216911:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539656.216912:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539656.216913:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539656.216915:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539656.216918:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539656.216920:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539737 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584539656.216929:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078448576 new x1661492078448624 00000800:00000100:0.0:1584539662.142896:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539662.143307:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584539662.155351:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539656/real 1584539656] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539743 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584539662.155360:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584539662.155361:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584539662.155362:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584539662.155368:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539662.155369:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539662.155370:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539662.155371:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539662.155372:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539662.155373:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584539662.155374:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584539662.155376:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21462 00000100:00080000:22.0:1584539662.155379:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584539662.155388:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539662.155389:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539662.155390:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539662.155391:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539662.155392:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539662.155393:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539662.155393:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539662.155394:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539662.202643:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539662.202647:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539662.202649:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539662.202650:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539662.202651:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539662.202653:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539662.202655:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539662.202656:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539743 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584539662.202686:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078448624 new x1661492078448672 00000800:00000100:0.0:1584539668.139907:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539668.140402:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584539668.152446:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539662/real 1584539662] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539749 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584539668.152454:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584539668.152455:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584539668.152456:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584539668.152471:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539668.152473:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539668.152474:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539668.152475:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539668.152476:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539668.152477:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584539668.152479:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584539668.152480:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21468 00000100:00080000:22.0:1584539668.152483:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584539668.152488:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539668.152490:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539668.152490:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539668.152491:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539668.152492:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539668.152493:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539668.152494:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539668.152495:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539668.206104:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539668.206108:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539668.206109:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539668.206110:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539668.206111:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539668.206113:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539668.206115:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539668.206116:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539749 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584539668.206125:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078448672 new x1661492078448720 00000800:00020000:2.0:1584539674.916039:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584539674.916042:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584539674.916044:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539674.916476:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584539674.928560:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539668/real 1584539668] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539755 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584539674.928568:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584539674.928569:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584539674.928570:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584539674.928576:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539674.928578:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539674.928578:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539674.928580:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539674.928581:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539674.928583:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584539674.928583:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584539674.928585:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21474 00000100:00080000:22.0:1584539674.928587:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584539674.928595:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539674.928597:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539674.928597:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539674.928599:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539674.928599:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539674.928600:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539674.928601:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539674.928602:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539674.971526:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539674.971530:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539674.971532:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539674.971533:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539674.971534:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539674.971536:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539674.971538:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539674.971540:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539755 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584539674.971569:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078448720 new x1661492078448768 00000800:00000100:0.0:1584539680.136654:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539680.137102:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584539680.149152:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539674/real 1584539674] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539761 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584539680.149160:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584539680.149161:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584539680.149162:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584539680.149168:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539680.149169:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539680.149178:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539680.149180:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539680.149180:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539680.149182:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584539680.149182:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584539680.149184:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21480 00000100:00080000:22.0:1584539680.149186:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584539680.149191:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539680.149193:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539680.149193:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539680.149194:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539680.149195:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539680.149196:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539680.149197:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539680.149198:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539680.212708:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539680.212712:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539680.212713:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539680.212714:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539680.212715:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539680.212717:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539680.212718:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539680.212720:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539761 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584539680.212728:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078448768 new x1661492078448816 00000800:00000100:0.0:1584539686.136451:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539686.136868:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584539686.148918:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539680/real 1584539680] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539767 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584539686.148925:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584539686.148927:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584539686.148928:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584539686.148934:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539686.148936:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539686.148937:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539686.148938:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539686.148939:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539686.148940:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584539686.148940:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584539686.148942:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21486 00000100:00080000:22.0:1584539686.148944:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584539686.148953:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539686.148954:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539686.148954:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539686.148956:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539686.148956:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539686.148957:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539686.148958:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539686.148959:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539686.188107:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539686.188110:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539686.188112:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539686.188113:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539686.188114:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539686.188116:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539686.188117:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539686.188119:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539767 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584539686.188127:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078448816 new x1661492078448848 00000800:00000100:0.0:1584539692.136396:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539692.136811:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584539692.148861:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539686/real 1584539686] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539773 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584539692.148869:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584539692.148870:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584539692.148872:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584539692.148887:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539692.148889:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539692.148889:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539692.148891:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539692.148891:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539692.148893:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584539692.148893:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584539692.148895:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21492 00000100:00080000:22.0:1584539692.148897:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584539692.148903:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539692.148904:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539692.148904:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539692.148905:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539692.148906:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539692.148907:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539692.148908:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539692.148909:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539692.182927:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539692.182931:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539692.182933:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539692.182934:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539692.182935:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539692.182937:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539692.182939:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539692.182941:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539773 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584539692.182971:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078448848 new x1661492078448912 00000800:00020000:2.0:1584539698.916095:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584539698.916101:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584539698.916105:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539698.916553:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584539698.928646:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539692/real 1584539692] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539779 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584539698.928666:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584539698.928667:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584539698.928668:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584539698.928674:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539698.928676:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539698.928677:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539698.928678:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539698.928678:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539698.928680:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584539698.928680:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584539698.928682:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21498 00000100:00080000:22.0:1584539698.928684:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584539698.928693:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539698.928694:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539698.928695:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539698.928696:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539698.928697:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539698.928698:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539698.928698:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539698.928700:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539698.991150:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539698.991155:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539698.991156:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539698.991157:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539698.991158:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539698.991160:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539698.991162:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539698.991164:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539779 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584539698.991193:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078448912 new x1661492078448944 00000800:00000100:0.0:1584539704.136543:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539704.136963:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584539704.149017:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539698/real 1584539698] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539785 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584539704.149042:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584539704.149043:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584539704.149045:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584539704.149053:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539704.149055:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539704.149066:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539704.149068:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539704.149069:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539704.149070:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584539704.149071:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584539704.149073:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21504 00000100:00080000:22.0:1584539704.149077:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584539704.149084:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539704.149086:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539704.149086:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539704.149088:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539704.149088:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539704.149090:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539704.149092:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539704.149093:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539704.196450:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539704.196456:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539704.196458:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539704.196459:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539704.196462:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539704.196464:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539704.196466:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539704.196468:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539785 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584539704.196477:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078448944 new x1661492078449008 00000800:00000100:0.0:1584539710.137048:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539710.137458:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584539710.149519:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539704/real 1584539704] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539791 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584539710.149529:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584539710.149531:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584539710.149533:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584539710.149541:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539710.149543:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539710.149544:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539710.149545:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539710.149546:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539710.149547:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584539710.149548:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584539710.149551:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21510 00000100:00080000:22.0:1584539710.149554:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584539710.149568:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539710.149569:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539710.149570:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539710.149571:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539710.149573:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539710.149575:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539710.149575:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539710.149576:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539710.202598:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539710.202605:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539710.202607:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539710.202608:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539710.202610:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539710.202613:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539710.202615:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539710.202617:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539791 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584539710.202626:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078449008 new x1661492078449040 00000800:00000100:0.0:1584539716.140368:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539716.140898:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584539716.152959:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539710/real 1584539710] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539797 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584539716.152970:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584539716.152972:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584539716.152974:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584539716.152997:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539716.152999:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539716.153000:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539716.153002:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539716.153002:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539716.153004:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584539716.153005:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584539716.153008:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21516 00000100:00080000:22.0:1584539716.153011:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584539716.153019:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539716.153021:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539716.153023:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539716.153024:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539716.153025:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539716.153039:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539716.153040:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539716.153041:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539716.195888:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539716.195894:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539716.195896:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539716.195898:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539716.195900:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539716.195902:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539716.195903:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539716.195905:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539797 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584539716.195916:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078449040 new x1661492078449088 00000800:00020000:2.0:1584539722.916055:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584539722.916058:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584539722.916060:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539722.916697:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584539722.928786:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539716/real 1584539716] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539803 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584539722.928807:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584539722.928809:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584539722.928811:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584539722.928819:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539722.928821:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539722.928822:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539722.928823:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539722.928824:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539722.928825:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584539722.928826:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584539722.928828:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21522 00000100:00080000:22.0:1584539722.928831:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584539722.928846:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539722.928847:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539722.928848:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539722.928849:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539722.928850:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539722.928851:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539722.928852:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539722.928853:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539722.965283:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539722.965294:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539722.965297:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539722.965300:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539722.965303:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539722.965308:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539722.965312:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539722.965316:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539803 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584539722.965364:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078449088 new x1661492078449136 00000800:00000100:0.0:1584539728.143291:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539728.143748:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584539728.155799:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539722/real 1584539722] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539809 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584539728.155807:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584539728.155808:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584539728.155810:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584539728.155816:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539728.155818:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539728.155827:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539728.155828:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539728.155829:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539728.155830:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584539728.155831:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584539728.155834:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21528 00000100:00080000:22.0:1584539728.155836:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584539728.155841:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539728.155843:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539728.155843:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539728.155844:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539728.155845:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539728.155846:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539728.155847:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539728.155848:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539728.205424:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539728.205428:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539728.205430:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539728.205431:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539728.205432:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539728.205434:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539728.205436:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539728.205438:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539809 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584539728.205468:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078449136 new x1661492078449184 00000800:00000100:0.0:1584539734.143022:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539734.143443:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584539734.155527:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539728/real 1584539728] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539815 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584539734.155535:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584539734.155536:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584539734.155537:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584539734.155543:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539734.155545:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539734.155546:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539734.155547:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539734.155548:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539734.155550:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584539734.155551:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584539734.155552:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21534 00000100:00080000:22.0:1584539734.155554:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584539734.155564:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539734.155565:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539734.155566:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539734.155567:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539734.155568:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539734.155569:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539734.155569:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539734.155571:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539734.196593:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539734.196598:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539734.196599:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539734.196600:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539734.196601:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539734.196603:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539734.196605:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539734.196606:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539815 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584539734.196637:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078449184 new x1661492078449232 00000800:00000100:0.0:1584539740.143282:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539740.143736:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584539740.155791:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539734/real 1584539734] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539821 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584539740.155799:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584539740.155800:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584539740.155802:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584539740.155815:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539740.155817:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539740.155818:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539740.155820:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539740.155821:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539740.155822:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584539740.155823:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584539740.155824:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21540 00000100:00080000:22.0:1584539740.155826:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584539740.155832:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539740.155833:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539740.155834:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539740.155835:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539740.155836:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539740.155837:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584539740.155837:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584539740.155839:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539740.204492:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539740.204496:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539740.204497:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539740.204498:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539740.204499:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539740.204501:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539740.204502:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539740.204504:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539821 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:0.0:1584539740.204514:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078449232 new x1661492078449280 00000800:00020000:2.0:1584539746.916072:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584539746.916074:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584539746.916076:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539746.916511:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584539746.928593:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539740/real 1584539740] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539827 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584539746.928612:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584539746.928613:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584539746.928614:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584539746.928620:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539746.928621:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539746.928622:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539746.928623:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539746.928625:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539746.928626:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584539746.928627:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584539746.928628:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21546 00000100:00080000:4.0:1584539746.928630:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584539746.928639:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539746.928640:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539746.928641:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539746.928642:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539746.928642:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539746.928643:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539746.928644:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539746.928645:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539746.972698:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539746.972707:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539746.972710:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539746.972712:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539746.972715:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539746.972719:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539746.972735:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539746.972737:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539827 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584539746.972769:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078449280 new x1661492078449328 00000800:00000100:0.0:1584539752.143301:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539752.143790:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584539752.155858:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539746/real 1584539746] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539833 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584539752.155866:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584539752.155867:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584539752.155868:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584539752.155875:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539752.155876:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539752.155885:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539752.155886:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539752.155887:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539752.155888:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584539752.155889:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584539752.155891:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21552 00000100:00080000:4.0:1584539752.155893:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584539752.155898:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539752.155899:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539752.155900:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539752.155901:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539752.155902:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539752.155903:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584539752.155903:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584539752.155905:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539752.213931:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539752.213935:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539752.213936:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539752.213937:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539752.213938:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539752.213940:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539752.213942:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539752.213943:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539833 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539752.213973:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078449328 new x1661492078449376 00000800:00000100:0.0:1584539758.143802:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539758.144299:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539758.156367:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539752/real 1584539752] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539839 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539758.156375:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539758.156376:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539758.156378:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539758.156384:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539758.156385:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539758.156386:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539758.156387:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539758.156388:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539758.156389:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539758.156390:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539758.156391:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21558 00000100:00080000:6.0:1584539758.156393:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539758.156403:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539758.156404:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539758.156405:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539758.156406:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539758.156406:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539758.156408:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539758.156408:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539758.156409:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539758.203296:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539758.203300:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539758.203301:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539758.203302:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539758.203303:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539758.203305:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539758.203307:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539758.203309:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539839 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539758.203338:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078449376 new x1661492078449408 00000800:00000100:0.0:1584539764.143701:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539764.144299:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539764.156376:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539758/real 1584539758] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539845 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539764.156391:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539764.156392:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539764.156394:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539764.156413:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539764.156415:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539764.156416:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539764.156417:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539764.156418:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539764.156420:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539764.156420:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539764.156423:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21564 00000100:00080000:6.0:1584539764.156427:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539764.156434:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539764.156435:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539764.156436:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539764.156437:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539764.156438:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539764.156439:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539764.156440:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539764.156441:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539764.208369:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539764.208376:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539764.208378:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539764.208379:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539764.208381:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539764.208384:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539764.208386:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539764.208388:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539845 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539764.208417:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078449408 new x1661492078449472 00000800:00020000:6.0:1584539770.916066:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584539770.916069:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584539770.916071:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539770.916529:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539770.928611:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539764/real 1584539764] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539851 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539770.928632:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539770.928634:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539770.928636:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539770.928643:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539770.928645:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539770.928646:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539770.928648:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539770.928648:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539770.928650:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539770.928651:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539770.928653:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21570 00000100:00080000:6.0:1584539770.928656:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539770.928671:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539770.928673:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539770.928673:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539770.928675:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539770.928676:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539770.928678:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539770.928689:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539770.928691:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539770.972985:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539770.972991:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539770.972994:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539770.972995:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539770.972997:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539770.972999:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539770.973001:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539770.973002:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539851 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539770.973036:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078449472 new x1661492078449504 00000800:00000100:0.0:1584539776.141943:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539776.142384:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539776.154450:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539770/real 1584539770] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539857 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539776.154460:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539776.154461:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539776.154463:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539776.154471:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539776.154473:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539776.154486:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539776.154488:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539776.154488:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539776.154490:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539776.154491:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539776.154493:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21576 00000100:00080000:6.0:1584539776.154496:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539776.154503:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539776.154504:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539776.154506:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539776.154507:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539776.154508:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539776.154509:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539776.154510:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539776.154511:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539776.197695:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539776.197702:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539776.197704:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539776.197706:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539776.197707:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539776.197710:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539776.197711:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539776.197713:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539857 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539776.197723:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078449504 new x1661492078449552 00000800:00000100:0.0:1584539782.139404:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539782.139804:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539782.151873:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539776/real 1584539776] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539863 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539782.151882:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539782.151883:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539782.151885:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539782.151891:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539782.151892:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539782.151893:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539782.151895:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539782.151896:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539782.151897:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539782.151898:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539782.151900:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21582 00000100:00080000:6.0:1584539782.151902:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539782.151913:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539782.151914:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539782.151915:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539782.151916:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539782.151917:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539782.151919:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539782.151919:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539782.151920:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539782.199365:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539782.199371:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539782.199372:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539782.199373:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539782.199375:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539782.199377:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539782.199380:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539782.199382:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539863 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539782.199411:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078449552 new x1661492078449600 00000800:00000100:0.0:1584539788.139527:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539788.139956:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539788.152042:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539782/real 1584539782] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539869 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539788.152051:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539788.152052:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539788.152053:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539788.152070:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539788.152071:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539788.152072:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539788.152074:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539788.152074:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539788.152076:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539788.152077:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539788.152080:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21588 00000100:00080000:6.0:1584539788.152082:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539788.152089:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539788.152090:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539788.152091:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539788.152092:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539788.152093:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539788.152094:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539788.152095:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539788.152096:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539788.188945:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539788.188950:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539788.188952:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539788.188953:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539788.188955:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539788.188957:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539788.188958:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539788.188960:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539869 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539788.188990:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078449600 new x1661492078449648 00000800:00020000:6.0:1584539794.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584539794.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584539794.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539794.916509:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539794.928594:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539788/real 1584539788] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539875 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539794.928601:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539794.928603:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539794.928604:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539794.928620:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539794.928622:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539794.928622:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539794.928624:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539794.928625:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539794.928626:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539794.928627:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539794.928628:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21594 00000100:00080000:6.0:1584539794.928631:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539794.928639:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539794.928640:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539794.928641:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539794.928642:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539794.928643:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539794.928644:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539794.928645:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539794.928646:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539794.936858:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539794.936863:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539794.936864:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539794.936865:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539794.936867:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539794.936869:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539794.936870:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539794.936872:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539875 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539794.936881:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078449648 new x1661492078449696 00000800:00000100:0.0:1584539800.139654:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539800.140206:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539800.152283:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539794/real 1584539794] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539881 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539800.152291:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539800.152292:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539800.152293:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539800.152299:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539800.152301:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539800.152310:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539800.152311:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539800.152312:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539800.152313:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539800.152313:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539800.152315:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21600 00000100:00080000:6.0:1584539800.152317:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539800.152322:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539800.152324:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539800.152324:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539800.152325:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539800.152326:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539800.152327:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539800.152328:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539800.152329:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539800.160253:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539800.160257:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539800.160258:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539800.160259:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539800.160260:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539800.160262:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539800.160264:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539800.160266:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539881 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539800.160295:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078449696 new x1661492078449744 00000800:00000100:0.0:1584539806.139728:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539806.140189:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539806.152257:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539800/real 1584539800] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539887 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539806.152265:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539806.152266:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539806.152268:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539806.152274:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539806.152275:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539806.152276:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539806.152277:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539806.152278:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539806.152279:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539806.152280:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539806.152281:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21606 00000100:00080000:6.0:1584539806.152283:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539806.152302:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539806.152303:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539806.152304:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539806.152305:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539806.152305:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539806.152306:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539806.152307:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539806.152308:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584539806.160338:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584539806.160342:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584539806.160343:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584539806.160344:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584539806.160345:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584539806.160347:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584539806.160348:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584539806.160350:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539887 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539806.160380:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078449744 new x1661492078449776 00000800:00000100:0.0:1584539812.139857:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539812.140286:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539812.152354:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539806/real 1584539806] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539893 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539812.152366:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539812.152367:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539812.152369:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539812.152385:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539812.152386:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539812.152387:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539812.152388:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539812.152389:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539812.152390:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539812.152391:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539812.152392:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21612 00000100:00080000:6.0:1584539812.152395:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539812.152401:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539812.152402:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539812.152403:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539812.152404:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539812.152404:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539812.152406:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539812.152406:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539812.152407:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584539812.167656:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584539812.167660:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584539812.167662:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584539812.167663:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584539812.167664:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584539812.167666:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584539812.167668:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584539812.167669:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539893 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539812.167698:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078449776 new x1661492078449840 00000800:00020000:4.0:1584539818.916068:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584539818.916071:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584539818.916073:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539818.916517:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539818.928594:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539812/real 1584539812] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539899 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539818.928602:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539818.928603:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539818.928604:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539818.928610:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539818.928611:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539818.928612:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539818.928613:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539818.928614:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539818.928615:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539818.928616:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539818.928617:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21618 00000100:00080000:6.0:1584539818.928619:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539818.928628:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539818.928629:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539818.928630:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539818.928631:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539818.928632:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539818.928633:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539818.928634:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539818.928635:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584539818.936168:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584539818.936172:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584539818.936173:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584539818.936174:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584539818.936175:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584539818.936177:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584539818.936179:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584539818.936181:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539899 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539818.936210:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078449840 new x1661492078449872 00000800:00000100:0.0:1584539824.139958:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539824.140403:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539824.152475:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539818/real 1584539818] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539905 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539824.152485:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539824.152486:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539824.152488:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539824.152495:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539824.152497:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539824.152507:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539824.152508:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539824.152509:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539824.152511:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539824.152512:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539824.152514:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21624 00000100:00080000:6.0:1584539824.152517:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539824.152524:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539824.152526:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539824.152526:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539824.152528:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539824.152529:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539824.152530:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539824.152531:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539824.152532:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584539824.160349:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584539824.160355:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584539824.160357:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584539824.160358:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584539824.160361:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584539824.160364:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584539824.160365:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584539824.160367:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539905 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539824.160377:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078449872 new x1661492078449936 00000800:00000100:0.0:1584539830.141033:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539830.141476:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539830.153547:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539824/real 1584539824] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539911 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539830.153557:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539830.153559:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539830.153561:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539830.153568:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539830.153569:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539830.153570:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539830.153572:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539830.153572:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539830.153574:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539830.153575:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539830.153577:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21630 00000100:00080000:6.0:1584539830.153580:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539830.153593:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539830.153594:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539830.153596:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539830.153597:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539830.153598:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539830.153599:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539830.153600:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539830.153601:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584539830.161401:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584539830.161408:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584539830.161410:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584539830.161411:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584539830.161413:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584539830.161415:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584539830.161417:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584539830.161419:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539911 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539830.161428:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078449936 new x1661492078449968 00000800:00000100:0.0:1584539836.141290:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539836.141722:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539836.153782:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539830/real 1584539830] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539917 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539836.153793:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539836.153794:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539836.153796:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539836.153817:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539836.153819:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539836.153820:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539836.153822:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539836.153823:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539836.153824:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539836.153825:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539836.153828:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21636 00000100:00080000:6.0:1584539836.153831:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539836.153840:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539836.153841:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539836.153842:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539836.153843:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539836.153844:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539836.153845:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539836.153846:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539836.153847:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539836.161974:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539836.161981:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539836.161984:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539836.161985:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539836.161987:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539836.161990:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539836.161991:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539836.161993:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539917 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539836.162002:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078449968 new x1661492078450016 00000800:00000100:0.0:1584539842.141283:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539842.141758:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539842.153811:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539836/real 1584539836] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539923 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539842.153821:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539842.153823:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539842.153824:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539842.153830:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539842.153832:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539842.153833:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539842.153835:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539842.153835:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539842.153837:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539842.153838:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539842.153840:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21642 00000100:00080000:6.0:1584539842.153843:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539842.153856:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539842.153857:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539842.153858:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539842.153859:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539842.153860:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539842.153861:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539842.153862:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539842.153863:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539842.161714:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539842.161720:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539842.161723:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539842.161724:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539842.161726:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539842.161728:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539842.161729:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539842.161731:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539923 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539842.161745:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078450016 new x1661492078450064 00000800:00000100:0.0:1584539848.141504:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539848.142092:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539848.154176:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539842/real 1584539842] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539929 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539848.154187:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539848.154189:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539848.154191:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539848.154199:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539848.154201:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539848.154220:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539848.154221:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539848.154222:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539848.154224:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539848.154225:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539848.154228:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21648 00000100:00080000:6.0:1584539848.154231:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539848.154238:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539848.154240:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539848.154240:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539848.154242:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539848.154243:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539848.154244:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539848.154245:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539848.154246:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539848.162517:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539848.162524:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539848.162526:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539848.162527:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539848.162529:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539848.162532:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539848.162534:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539848.162536:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539929 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539848.162545:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078450064 new x1661492078450112 00000800:00000100:0.0:1584539854.141405:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539854.141865:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539854.153939:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539848/real 1584539848] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539935 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539854.153948:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539854.153949:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539854.153951:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539854.153957:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539854.153958:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539854.153959:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539854.153961:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539854.153962:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539854.153963:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539854.153964:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539854.153966:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21654 00000100:00080000:6.0:1584539854.153968:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539854.153978:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539854.153980:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539854.153980:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539854.153982:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539854.153982:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539854.153983:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539854.153984:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539854.153985:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539854.162013:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539854.162019:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539854.162021:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539854.162022:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539854.162023:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539854.162025:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539854.162039:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539854.162041:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539935 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539854.162071:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078450112 new x1661492078450160 00000800:00000100:0.0:1584539860.141434:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539860.141859:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539860.153927:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539854/real 1584539854] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539941 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539860.153934:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539860.153936:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539860.153937:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539860.153951:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539860.153954:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539860.153955:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539860.153956:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539860.153956:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539860.153958:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539860.153958:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539860.153960:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21660 00000100:00080000:6.0:1584539860.153962:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539860.153967:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539860.153968:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539860.153969:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539860.153970:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539860.153970:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539860.153972:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539860.153972:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539860.153973:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539860.162188:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539860.162192:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539860.162193:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539860.162194:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539860.162195:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539860.162197:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539860.162199:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539860.162200:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539941 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539860.162211:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078450160 new x1661492078450208 00000800:00000100:0.0:1584539866.141830:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539866.142328:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539866.154376:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539860/real 1584539860] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539947 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539866.154384:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539866.154385:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539866.154387:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539866.154391:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539866.154393:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539866.154394:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539866.154396:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539866.154396:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539866.154397:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539866.154398:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539866.154400:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21666 00000100:00080000:6.0:1584539866.154402:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539866.154410:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539866.154411:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539866.154412:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539866.154413:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539866.154414:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539866.154415:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539866.154415:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539866.154417:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539866.158243:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539866.158247:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539866.158248:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539866.158249:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539866.158251:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539866.158252:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539866.158254:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539866.158256:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539947 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539866.158264:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078450208 new x1661492078450240 00000800:00000100:0.0:1584539872.142686:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539872.143105:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539872.155156:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539866/real 1584539866] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539953 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539872.155168:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539872.155169:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539872.155171:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539872.155176:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539872.155177:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539872.155187:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539872.155188:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539872.155189:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539872.155190:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539872.155191:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539872.155193:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21672 00000100:00080000:6.0:1584539872.155194:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539872.155200:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539872.155202:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539872.155202:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539872.155203:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539872.155204:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539872.155205:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539872.155206:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539872.155207:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539872.159256:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539872.159260:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539872.159261:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539872.159262:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539872.159264:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539872.159266:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539872.159267:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539872.159269:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539953 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539872.159300:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078450240 new x1661492078450304 00000800:00000100:0.0:1584539878.141723:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539878.142243:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539878.154294:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539872/real 1584539872] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539959 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539878.154301:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539878.154302:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539878.154304:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539878.154309:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539878.154310:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539878.154311:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539878.154312:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539878.154313:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539878.154314:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539878.154315:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539878.154316:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21678 00000100:00080000:6.0:1584539878.154318:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539878.154327:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539878.154329:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539878.154329:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539878.154330:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539878.154331:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539878.154332:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539878.154333:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539878.154335:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539878.158372:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539878.158376:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539878.158377:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539878.158378:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539878.158379:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539878.158381:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539878.158383:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539878.158385:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539959 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584539878.158395:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078450304 new x1661492078450336 00000800:00000100:0.0:1584539884.141751:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539884.142175:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584539884.154250:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539878/real 1584539878] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539965 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584539884.154260:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584539884.154262:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584539884.154264:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584539884.154286:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539884.154288:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539884.154288:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539884.154290:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539884.154291:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539884.154293:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584539884.154294:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584539884.154296:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21684 00000100:00080000:6.0:1584539884.154299:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584539884.154307:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539884.154308:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539884.154309:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539884.154310:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539884.154312:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539884.154314:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584539884.154314:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584539884.154316:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539884.158311:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539884.158319:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539884.158321:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539884.158323:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539884.158325:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539884.158328:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539884.158329:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539884.158331:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539965 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539884.158361:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078450336 new x1661492078450400 00000800:00000100:0.0:1584539890.141281:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539890.141723:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539890.153768:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539884/real 1584539884] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539971 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539890.153777:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539890.153779:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539890.153780:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539890.153786:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539890.153788:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539890.153789:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539890.153790:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539890.153792:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539890.153794:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539890.153795:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539890.153797:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21690 00000100:00080000:8.0:1584539890.153799:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539890.153815:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539890.153816:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539890.153817:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539890.153818:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539890.153819:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539890.153820:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539890.153821:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539890.153822:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539890.157715:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539890.157721:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539890.157723:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539890.157725:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539890.157726:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539890.157729:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539890.157730:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539890.157732:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539971 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584539890.157762:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078450400 new x1661492078450432 00000800:00000100:0.0:1584539896.140464:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539896.140930:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584539896.152978:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539890/real 1584539890] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539977 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584539896.152987:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584539896.152988:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584539896.152990:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584539896.152996:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539896.152997:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539896.153011:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539896.153012:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539896.153013:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539896.153014:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584539896.153015:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584539896.153018:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21696 00000100:00080000:8.0:1584539896.153020:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584539896.153063:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539896.153065:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539896.153066:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539896.153067:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539896.153068:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539896.153069:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584539896.153070:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584539896.153071:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539896.156978:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539896.156984:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539896.156986:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539896.156987:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539896.156989:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539896.156991:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539896.156993:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539896.156995:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539977 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539896.157006:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078450432 new x1661492078450480 00000800:00000100:0.0:1584539902.140564:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539902.140975:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539902.153054:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539896/real 1584539896] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539983 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539902.153062:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539902.153063:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539902.153065:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539902.153071:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539902.153073:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539902.153074:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539902.153075:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539902.153076:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539902.153077:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539902.153078:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539902.153079:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21702 00000100:00080000:10.0:1584539902.153082:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539902.153094:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539902.153097:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539902.153098:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539902.153099:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539902.153099:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539902.153101:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539902.153101:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539902.153102:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539902.156997:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539902.157002:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539902.157004:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539902.157005:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539902.157006:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539902.157008:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539902.157010:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539902.157011:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539983 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539902.157020:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078450480 new x1661492078450528 00000800:00000100:0.0:1584539908.140460:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539908.140875:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539908.152924:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539902/real 1584539902] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539989 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539908.152932:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539908.152933:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539908.152935:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539908.152949:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539908.152951:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539908.152952:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539908.152953:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539908.152953:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539908.152955:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539908.152955:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539908.152957:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21708 00000100:00080000:10.0:1584539908.152960:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539908.152965:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539908.152967:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539908.152967:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539908.152969:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539908.152970:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539908.152971:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539908.152971:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539908.152972:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539908.156600:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539908.156604:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539908.156606:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539908.156606:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539908.156608:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539908.156610:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539908.156611:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539908.156613:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539989 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539908.156621:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078450528 new x1661492078450576 00000800:00000100:0.0:1584539914.140531:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539914.141034:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539914.153080:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539908/real 1584539908] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539995 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539914.153088:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539914.153089:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539914.153090:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539914.153095:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539914.153096:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539914.153097:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539914.153098:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539914.153099:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539914.153100:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539914.153102:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539914.153103:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21714 00000100:00080000:10.0:1584539914.153105:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539914.153114:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539914.153115:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539914.153116:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539914.153117:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539914.153117:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539914.153118:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539914.153119:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539914.153120:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539914.157022:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539914.157026:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539914.157039:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539914.157040:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539914.157042:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539914.157044:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539914.157045:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539914.157047:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584539995 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539914.157079:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078450576 new x1661492078450624 00000800:00000100:0.0:1584539920.141685:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539920.142149:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539920.154201:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539914/real 1584539914] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540001 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539920.154209:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539920.154210:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539920.154211:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539920.154217:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539920.154218:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539920.154226:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539920.154227:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539920.154229:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539920.154230:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539920.154231:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539920.154233:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21720 00000100:00080000:10.0:1584539920.154234:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539920.154240:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539920.154241:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539920.154242:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539920.154243:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539920.154244:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539920.154245:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539920.154245:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539920.154246:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539920.158112:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539920.158116:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539920.158118:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539920.158118:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539920.158120:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539920.158122:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539920.158123:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539920.158125:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540001 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539920.158133:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078450624 new x1661492078450672 00000800:00000100:0.0:1584539926.140782:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539926.141198:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539926.153249:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539920/real 1584539920] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540007 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539926.153256:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539926.153257:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539926.153258:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539926.153263:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539926.153265:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539926.153266:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539926.153268:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539926.153268:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539926.153270:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539926.153270:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539926.153272:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21726 00000100:00080000:10.0:1584539926.153274:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539926.153281:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539926.153283:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539926.153283:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539926.153284:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539926.153285:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539926.153286:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539926.153287:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539926.153288:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539926.156992:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539926.156996:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539926.156997:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539926.156998:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539926.156999:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539926.157001:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539926.157003:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539926.157004:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540007 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539926.157012:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078450672 new x1661492078450704 00000800:00000100:0.0:1584539932.140812:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539932.141252:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539932.153294:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539926/real 1584539926] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540013 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539932.153301:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539932.153302:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539932.153303:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539932.153317:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539932.153319:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539932.153319:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539932.153321:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539932.153321:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539932.153323:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539932.153323:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539932.153325:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21732 00000100:00080000:10.0:1584539932.153327:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539932.153333:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539932.153334:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539932.153334:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539932.153336:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539932.153336:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539932.153337:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539932.153338:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539932.153339:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539932.157424:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539932.157428:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539932.157429:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539932.157430:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539932.157431:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539932.157433:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539932.157435:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539932.157436:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540013 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539932.157465:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078450704 new x1661492078450768 00000800:00000100:0.0:1584539938.140908:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539938.141401:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539938.153451:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539932/real 1584539932] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540019 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539938.153459:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539938.153461:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539938.153462:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539938.153467:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539938.153468:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539938.153469:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539938.153470:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539938.153471:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539938.153472:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539938.153473:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539938.153475:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21738 00000100:00080000:10.0:1584539938.153477:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539938.153485:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539938.153487:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539938.153487:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539938.153488:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539938.153489:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539938.153490:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539938.153491:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539938.153492:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539938.157272:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539938.157276:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539938.157278:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539938.157278:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539938.157280:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539938.157282:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539938.157283:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539938.157285:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540019 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539938.157314:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078450768 new x1661492078450800 00000800:00020000:2.0:1584539944.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584539944.938891:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584539944.963065:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539944.963572:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539944.975629:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539938/real 1584539938] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540025 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539944.975639:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539944.975640:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539944.975642:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539944.975648:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539944.975650:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539944.975660:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539944.975661:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539944.975662:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539944.975664:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539944.975665:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539944.975667:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21744 00000100:00080000:10.0:1584539944.975669:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539944.975676:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539944.975677:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539944.975678:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539944.975679:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539944.975680:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539944.975681:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539944.975682:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539944.975683:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539944.979724:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539944.979729:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539944.979731:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539944.979733:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539944.979734:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539944.979736:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539944.979738:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539944.979740:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540025 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539944.979769:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078450800 new x1661492078450864 00000800:00000100:0.0:1584539950.141035:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539950.141472:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539950.153542:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539944/real 1584539944] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540031 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539950.153552:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539950.153553:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539950.153555:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539950.153563:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539950.153565:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539950.153566:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539950.153567:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539950.153568:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539950.153569:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539950.153571:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539950.153573:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21750 00000100:00080000:10.0:1584539950.153575:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539950.153591:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539950.153592:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539950.153593:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539950.153594:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539950.153595:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539950.153596:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539950.153598:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539950.153599:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539950.157562:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539950.157569:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539950.157571:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539950.157572:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539950.157574:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539950.157576:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539950.157578:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539950.157580:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540031 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539950.157609:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078450864 new x1661492078450896 00000800:00000100:0.0:1584539956.140612:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539956.141103:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539956.153160:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539950/real 1584539950] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540037 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539956.153170:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539956.153172:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539956.153174:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539956.153197:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539956.153199:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539956.153200:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539956.153201:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539956.153202:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539956.153204:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539956.153205:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539956.153207:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21756 00000100:00080000:10.0:1584539956.153210:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539956.153218:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539956.153219:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539956.153220:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539956.153221:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539956.153223:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539956.153224:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539956.153225:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539956.153226:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539956.157206:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539956.157213:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539956.157215:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539956.157217:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539956.157219:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539956.157221:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539956.157223:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539956.157225:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540037 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539956.157233:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078450896 new x1661492078450944 00000800:00000100:0.0:1584539962.140700:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539962.141160:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539962.153233:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539956/real 1584539956] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540043 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539962.153242:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539962.153243:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539962.153244:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539962.153251:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539962.153253:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539962.153254:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539962.153255:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539962.153256:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539962.153257:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539962.153258:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539962.153260:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21762 00000100:00080000:10.0:1584539962.153263:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539962.153274:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539962.153275:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539962.153276:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539962.153278:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539962.153278:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539962.153280:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539962.153280:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539962.153281:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539962.157358:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539962.157364:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539962.157366:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539962.157367:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539962.157369:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539962.157372:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539962.157373:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539962.157375:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540043 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539962.157404:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078450944 new x1661492078450992 00000800:00020000:6.0:1584539968.916081:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584539968.916084:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584539968.916086:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539968.916470:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539968.928525:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539962/real 1584539962] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540049 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539968.928534:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539968.928535:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539968.928537:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539968.928543:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539968.928545:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539968.928555:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539968.928556:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539968.928557:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539968.928559:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539968.928559:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539968.928561:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21768 00000100:00080000:10.0:1584539968.928564:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539968.928571:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539968.928572:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539968.928573:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539968.928574:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539968.928575:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539968.928577:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539968.928577:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539968.928578:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539968.932520:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539968.932526:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539968.932529:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539968.932530:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539968.932531:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539968.932533:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539968.932535:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539968.932537:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540049 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539968.932566:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078450992 new x1661492078451040 00000800:00000100:0.0:1584539974.140808:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539974.141300:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539974.153380:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539968/real 1584539968] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540055 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539974.153388:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539974.153389:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539974.153391:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539974.153397:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539974.153399:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539974.153400:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539974.153401:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539974.153402:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539974.153403:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539974.153404:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539974.153407:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21774 00000100:00080000:10.0:1584539974.153409:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539974.153418:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539974.153419:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539974.153420:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539974.153421:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539974.153422:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539974.153423:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539974.153424:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539974.153425:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539974.157393:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539974.157398:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539974.157400:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539974.157401:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539974.157403:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539974.157405:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539974.157406:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539974.157408:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540055 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539974.157437:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078451040 new x1661492078451088 00000800:00000100:0.0:1584539980.140947:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539980.141411:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539980.153487:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539974/real 1584539974] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540061 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539980.153495:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539980.153496:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539980.153497:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539980.153511:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539980.153513:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539980.153514:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539980.153515:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539980.153516:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539980.153520:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539980.153521:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539980.153523:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21780 00000100:00080000:10.0:1584539980.153525:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539980.153530:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539980.153531:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539980.153532:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539980.153533:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539980.153534:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539980.153535:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539980.153535:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539980.153537:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539980.157409:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539980.157413:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539980.157415:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539980.157415:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539980.157417:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539980.157419:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539980.157420:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539980.157422:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540061 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539980.157451:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078451088 new x1661492078451136 00000800:00000100:0.0:1584539986.141018:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539986.141763:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539986.153839:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539980/real 1584539980] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540067 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539986.153847:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539986.153848:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539986.153849:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539986.153855:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539986.153857:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539986.153857:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539986.153860:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539986.153860:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539986.153862:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539986.153862:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539986.153864:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21786 00000100:00080000:10.0:1584539986.153866:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539986.153874:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539986.153875:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539986.153876:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539986.153877:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539986.153878:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539986.153879:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539986.153879:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539986.153880:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539986.157895:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539986.157899:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539986.157900:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539986.157901:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539986.157902:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539986.157904:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539986.157905:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539986.157907:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540067 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539986.157937:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078451136 new x1661492078451168 00000800:00020000:4.0:1584539992.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584539992.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584539992.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584539992.916498:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539992.928549:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539986/real 1584539986] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540073 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539992.928557:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539992.928558:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539992.928559:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539992.928565:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539992.928567:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539992.928576:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539992.928577:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539992.928578:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539992.928580:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539992.928580:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539992.928582:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21792 00000100:00080000:10.0:1584539992.928584:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539992.928590:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539992.928591:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539992.928592:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539992.928593:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539992.928594:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539992.928595:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539992.928595:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539992.928597:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539992.932709:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539992.932713:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539992.932715:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539992.932715:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539992.932717:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539992.932719:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539992.932720:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539992.932722:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540073 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539992.932750:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078451168 new x1661492078451232 00000800:00000100:0.0:1584539998.141218:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584539998.141711:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584539998.153780:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539992/real 1584539992] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540079 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584539998.153788:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584539998.153789:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584539998.153792:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584539998.153798:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539998.153799:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539998.153800:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539998.153801:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539998.153802:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539998.153803:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584539998.153804:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584539998.153805:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21798 00000100:00080000:10.0:1584539998.153807:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584539998.153816:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539998.153817:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539998.153818:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539998.153819:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539998.153820:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539998.153821:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584539998.153822:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584539998.153823:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584539998.157955:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584539998.157960:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584539998.157961:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584539998.157962:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584539998.157963:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584539998.157965:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584539998.157967:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584539998.157968:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540079 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584539998.157997:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078451232 new x1661492078451264 00000002:00080000:8.0:1584540002.275170:0:175414:0:(mdc_request.c:903:mdc_close()) @@@ matched open req@ffff8d3bbf9b0000 x1661492078451328/t87770749597(87770749597) o101->public1-MDT0000-mdc-ffff8d3ae8ab1800@10.10.2.11@o2ib:12/10 lens 576/616 e 0 to 0 dl 1584540011 ref 1 fl Complete:RP/4/ffffffff rc 0/-1 00000800:00000100:0.0:1584540004.141238:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540004.141683:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540004.153748:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584539998/real 1584539998] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540085 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540004.153762:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540004.153764:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540004.153766:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540004.153783:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540004.153785:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540004.153786:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540004.153787:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540004.153788:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540004.153789:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540004.153790:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540004.153793:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21804 00000100:00080000:10.0:1584540004.153795:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584540004.153803:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540004.153805:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540004.153805:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540004.153807:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540004.153807:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540004.153809:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540004.153809:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540004.153811:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540004.157870:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584540004.157876:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584540004.157878:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584540004.157879:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584540004.157881:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584540004.157883:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584540004.157885:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584540004.157887:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540085 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584540004.157918:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078451264 new x1661492078451424 00000002:00080000:6.0:1584540006.995757:0:176010:0:(mdc_request.c:903:mdc_close()) @@@ matched open req@ffff8d3397591f80 x1661492078451456/t87770775366(87770775366) o101->public1-MDT0000-mdc-ffff8d3ae8ab1800@10.10.2.11@o2ib:12/10 lens 576/616 e 0 to 0 dl 1584540016 ref 1 fl Complete:RP/4/ffffffff rc 0/-1 00000800:00000100:0.0:1584540010.140474:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540010.140880:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540010.152951:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540004/real 1584540004] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540091 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540010.152960:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540010.152961:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540010.152963:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540010.152977:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540010.152979:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540010.152980:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540010.152981:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540010.152982:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540010.152984:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540010.152985:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540010.152987:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21810 00000100:00080000:10.0:1584540010.152989:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584540010.152996:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540010.152998:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540010.152998:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540010.153000:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540010.153000:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540010.153002:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540010.153003:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540010.153005:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540010.157068:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584540010.157074:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584540010.157076:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584540010.157077:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584540010.157079:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584540010.157081:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584540010.157083:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584540010.157085:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540091 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584540010.157114:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078451424 new x1661492078451536 00000800:00020000:2.0:1584540016.916060:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584540016.916063:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (6): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584540016.916065:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540016.916522:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540016.928604:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540010/real 1584540010] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540097 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540016.928614:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540016.928615:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540016.928617:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540016.928625:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540016.928627:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540016.928639:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540016.928640:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540016.928641:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540016.928643:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540016.928643:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540016.928646:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21816 00000100:00080000:10.0:1584540016.928648:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584540016.928655:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540016.928657:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540016.928658:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540016.928659:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540016.928661:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540016.928662:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540016.928663:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540016.928664:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540016.932606:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584540016.932612:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584540016.932614:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584540016.932616:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584540016.932618:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584540016.932620:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584540016.932622:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584540016.932624:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540097 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:0.0:1584540016.932634:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078451536 new x1661492078451600 00000800:00000100:0.0:1584540022.140675:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540022.141118:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584540022.153190:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540016/real 1584540016] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540103 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584540022.153198:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584540022.153200:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584540022.153201:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584540022.153207:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540022.153209:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540022.153210:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540022.153211:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540022.153212:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540022.153214:0:4532:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584540022.153214:0:4532:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584540022.153216:0:4532:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21822 00000100:00080000:2.0:1584540022.153218:0:4532:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584540022.153228:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540022.153229:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540022.153229:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540022.153231:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540022.153231:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540022.153232:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540022.153233:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540022.153234:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540022.157317:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584540022.157321:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584540022.157323:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584540022.157324:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584540022.157325:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584540022.157327:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584540022.157328:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584540022.157330:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540103 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584540022.157340:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078451600 new x1661492078451632 00000800:00000100:0.0:1584540028.140740:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540028.141193:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584540028.153263:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540022/real 1584540022] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540109 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584540028.153270:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584540028.153272:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584540028.153273:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584540028.153286:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540028.153288:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540028.153289:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540028.153290:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540028.153291:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540028.153292:0:4532:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584540028.153293:0:4532:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584540028.153294:0:4532:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21828 00000100:00080000:2.0:1584540028.153296:0:4532:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584540028.153302:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540028.153303:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540028.153304:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540028.153305:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540028.153305:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540028.153307:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540028.153307:0:4532:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540028.153308:0:4532:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540028.157371:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540028.157375:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540028.157377:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540028.157378:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540028.157379:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540028.157381:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540028.157382:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540028.157385:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540109 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584540028.157415:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078451632 new x1661492078451680 00000800:00000100:0.0:1584540034.140827:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540034.141324:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584540034.153394:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540028/real 1584540028] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540115 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584540034.153403:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584540034.153404:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584540034.153405:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584540034.153411:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540034.153412:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540034.153413:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540034.153414:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540034.153415:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540034.153416:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584540034.153417:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584540034.153418:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21834 00000100:00080000:4.0:1584540034.153420:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584540034.153429:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540034.153431:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540034.153431:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540034.153432:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540034.153433:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540034.153434:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540034.153435:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540034.153437:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540034.157330:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540034.157334:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540034.157335:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540034.157336:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540034.157337:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540034.157339:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540034.157341:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540034.157343:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540115 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584540034.157370:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078451680 new x1661492078451728 00000800:00020000:4.0:1584540040.916175:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584540040.916178:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584540040.916180:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540040.916623:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584540040.928673:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540034/real 1584540034] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540121 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584540040.928680:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584540040.928681:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584540040.928682:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584540040.928687:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540040.928688:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540040.928697:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540040.928698:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540040.928699:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540040.928700:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584540040.928701:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584540040.928702:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21840 00000100:00080000:4.0:1584540040.928704:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584540040.928710:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540040.928711:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540040.928711:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540040.928714:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540040.928714:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540040.928715:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540040.928716:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540040.928717:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540040.932647:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540040.932651:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540040.932652:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540040.932653:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540040.932654:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540040.932656:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540040.932658:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540040.932659:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540121 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584540040.932690:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078451728 new x1661492078451776 00000800:00000100:0.0:1584540046.140906:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540046.141305:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584540046.153353:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540040/real 1584540040] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540127 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584540046.153360:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584540046.153361:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584540046.153362:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584540046.153367:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540046.153368:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540046.153369:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540046.153370:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540046.153371:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540046.153372:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584540046.153373:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584540046.153375:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21846 00000100:00080000:4.0:1584540046.153377:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584540046.153385:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540046.153387:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540046.153387:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540046.153388:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540046.153389:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540046.153390:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540046.153391:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540046.153392:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540046.157292:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540046.157296:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540046.157298:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540046.157299:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540046.157300:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540046.157302:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540046.157304:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540046.157306:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540127 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584540046.157339:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078451776 new x1661492078451808 00000800:00000100:0.0:1584540052.141041:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540052.141466:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584540052.153533:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540046/real 1584540046] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540133 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584540052.153541:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584540052.153542:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584540052.153544:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584540052.153558:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540052.153560:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540052.153560:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540052.153562:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540052.153562:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540052.153564:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584540052.153564:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584540052.153566:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21852 00000100:00080000:4.0:1584540052.153569:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584540052.153574:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540052.153576:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540052.153576:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540052.153577:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540052.153578:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540052.153579:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540052.153580:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540052.153581:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540052.157473:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540052.157477:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540052.157478:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540052.157479:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540052.157481:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540052.157482:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540052.157484:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540052.157486:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540133 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584540052.157514:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078451808 new x1661492078451872 00000800:00000100:0.0:1584540058.141466:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540058.141939:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584540058.154008:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540052/real 1584540052] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540139 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584540058.154015:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584540058.154017:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584540058.154018:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584540058.154024:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540058.154025:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540058.154031:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540058.154033:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540058.154033:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540058.154035:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584540058.154036:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584540058.154038:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21858 00000100:00080000:4.0:1584540058.154040:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584540058.154049:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540058.154051:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540058.154051:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540058.154053:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540058.154053:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540058.154054:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540058.154055:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540058.154056:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540058.157925:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540058.157929:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540058.157931:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540058.157932:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540058.157933:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540058.157935:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540058.157936:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540058.157938:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540139 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584540058.157948:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078451872 new x1661492078451904 00000800:00020000:2.0:1584540064.916049:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584540064.916052:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584540064.916054:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540064.916437:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584540064.928488:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540058/real 1584540058] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540145 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584540064.928496:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584540064.928497:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584540064.928498:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584540064.928504:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540064.928506:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540064.928515:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540064.928516:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540064.928518:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540064.928519:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584540064.928520:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584540064.928522:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21864 00000100:00080000:4.0:1584540064.928524:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584540064.928529:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540064.928530:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540064.928531:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540064.928532:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540064.928533:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540064.928534:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540064.928535:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540064.928536:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540064.932359:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540064.932364:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540064.932366:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540064.932367:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540064.932368:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540064.932370:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540064.932373:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540064.932374:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540145 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584540064.932403:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078451904 new x1661492078451968 00000800:00000100:0.0:1584540070.141393:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540070.141793:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584540070.153848:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540064/real 1584540064] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540151 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584540070.153856:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584540070.153857:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584540070.153858:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584540070.153863:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540070.153864:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540070.153866:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540070.153867:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540070.153868:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540070.153869:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584540070.153870:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584540070.153872:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21870 00000100:00080000:4.0:1584540070.153874:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584540070.153882:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540070.153883:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540070.153883:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540070.153885:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540070.153885:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540070.153886:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540070.153887:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540070.153888:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540070.157908:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540070.157912:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540070.157913:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540070.157914:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540070.157915:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540070.157917:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540070.157918:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540070.157920:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540151 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584540070.157950:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078451968 new x1661492078452000 00000800:00000100:0.0:1584540076.141433:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540076.141954:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584540076.154025:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540070/real 1584540070] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540157 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584540076.154041:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584540076.154043:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584540076.154045:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584540076.154067:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540076.154069:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540076.154070:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540076.154072:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540076.154072:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540076.154074:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584540076.154075:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584540076.154077:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21876 00000100:00080000:4.0:1584540076.154081:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584540076.154088:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540076.154090:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540076.154090:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540076.154092:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540076.154092:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540076.154094:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540076.154094:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540076.154096:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540076.158099:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540076.158105:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540076.158108:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540076.158109:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540076.158111:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540076.158113:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540076.158115:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540076.158117:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540157 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584540076.158127:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078452000 new x1661492078452048 00000800:00000100:0.0:1584540082.140378:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540082.140792:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584540082.152869:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540076/real 1584540076] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540163 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584540082.152880:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584540082.152883:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584540082.152885:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584540082.152893:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540082.152895:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540082.152896:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540082.152897:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540082.152898:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540082.152900:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584540082.152901:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584540082.152903:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21882 00000100:00080000:4.0:1584540082.152907:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584540082.152923:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540082.152925:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540082.152925:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540082.152927:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540082.152927:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540082.152929:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540082.152929:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584540082.152931:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540082.157049:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540082.157056:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540082.157059:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540082.157060:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540082.157062:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540082.157064:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540082.157067:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540082.157069:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540163 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584540082.157099:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078452048 new x1661492078452096 00000800:00020000:2.0:1584540088.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584540088.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584540088.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540088.916488:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584540088.928545:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540082/real 1584540082] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540169 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584540088.928553:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584540088.928554:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584540088.928555:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584540088.928561:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540088.928563:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540088.928573:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540088.928574:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540088.928575:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540088.928577:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584540088.928577:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584540088.928579:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21888 00000100:00080000:6.0:1584540088.928582:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584540088.928588:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540088.928589:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540088.928590:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540088.928591:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540088.928592:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540088.928593:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540088.928594:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540088.928595:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540088.932813:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540088.932819:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540088.932820:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540088.932821:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540088.932823:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540088.932825:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540088.932827:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540088.932828:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540169 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584540088.932836:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078452096 new x1661492078452144 00000800:00000100:0.0:1584540094.140495:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540094.140962:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584540094.153012:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540088/real 1584540088] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540175 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584540094.153020:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584540094.153021:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584540094.153023:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584540094.153036:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540094.153037:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540094.153038:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540094.153039:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540094.153040:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540094.153041:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584540094.153045:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584540094.153047:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21894 00000100:00080000:6.0:1584540094.153049:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584540094.153057:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540094.153059:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540094.153059:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540094.153060:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540094.153061:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540094.153062:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540094.153063:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540094.153064:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540094.156935:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540094.156939:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540094.156941:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540094.156942:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540094.156943:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540094.156945:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540094.156947:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540094.156949:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540175 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584540094.156979:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078452144 new x1661492078452192 00000800:00000100:0.0:1584540100.140527:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540100.140930:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584540100.152980:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540094/real 1584540094] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540181 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584540100.152988:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584540100.152989:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584540100.152990:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584540100.153002:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540100.153004:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540100.153005:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540100.153006:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540100.153007:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540100.153009:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584540100.153009:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584540100.153011:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21900 00000100:00080000:6.0:1584540100.153013:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584540100.153019:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540100.153020:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540100.153020:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540100.153022:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540100.153022:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540100.153023:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540100.153024:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540100.153025:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540100.157098:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540100.157102:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540100.157103:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540100.157104:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540100.157105:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540100.157107:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540100.157109:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540100.157111:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540181 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584540100.157141:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078452192 new x1661492078452240 00000800:00000100:0.0:1584540106.140674:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540106.141094:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584540106.153141:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540100/real 1584540100] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540187 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584540106.153149:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584540106.153150:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584540106.153151:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584540106.153156:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540106.153157:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540106.153159:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540106.153160:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540106.153161:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540106.153162:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584540106.153163:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584540106.153164:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21906 00000100:00080000:6.0:1584540106.153166:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584540106.153184:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540106.153185:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540106.153186:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540106.153187:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540106.153188:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540106.153189:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540106.153189:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540106.153190:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540106.157245:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540106.157249:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540106.157251:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540106.157252:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540106.157253:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540106.157255:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540106.157256:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540106.157258:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540187 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584540106.157267:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078452240 new x1661492078452272 00000800:00020000:4.0:1584540112.916051:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584540112.916054:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584540112.916056:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540112.916470:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584540112.928521:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540106/real 1584540106] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540193 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584540112.928529:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584540112.928530:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584540112.928531:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584540112.928537:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540112.928539:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540112.928548:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540112.928550:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540112.928550:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540112.928552:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584540112.928552:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584540112.928554:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21912 00000100:00080000:6.0:1584540112.928556:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584540112.928561:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540112.928563:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540112.928563:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540112.928564:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540112.928565:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540112.928566:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540112.928567:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540112.928568:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540112.932295:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540112.932299:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540112.932300:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540112.932301:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540112.932302:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540112.932304:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540112.932306:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540112.932308:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540193 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584540112.932317:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078452272 new x1661492078452336 00000800:00000100:0.0:1584540118.140902:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540118.141317:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584540118.153366:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540112/real 1584540112] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540199 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584540118.153374:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584540118.153376:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584540118.153377:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584540118.153381:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540118.153383:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540118.153384:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540118.153385:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540118.153385:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540118.153387:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584540118.153387:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584540118.153389:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21918 00000100:00080000:6.0:1584540118.153391:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584540118.153400:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540118.153401:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540118.153402:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540118.153403:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540118.153403:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540118.153405:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540118.153405:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540118.153406:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540118.157403:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540118.157407:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540118.157408:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540118.157409:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540118.157410:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540118.157412:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540118.157415:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540118.157417:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540199 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584540118.157425:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078452336 new x1661492078452368 00000800:00000100:0.0:1584540124.140949:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540124.141334:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584540124.153386:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540118/real 1584540118] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540205 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584540124.153396:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584540124.153397:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584540124.153398:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584540124.153415:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540124.153417:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540124.153417:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540124.153419:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540124.153419:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540124.153421:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584540124.153421:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584540124.153423:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21924 00000100:00080000:6.0:1584540124.153426:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584540124.153432:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540124.153433:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540124.153434:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540124.153435:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540124.153436:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540124.153437:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540124.153438:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540124.153439:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540124.157496:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540124.157501:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540124.157502:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540124.157503:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540124.157505:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540124.157507:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540124.157509:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540124.157511:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540205 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584540124.157519:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078452368 new x1661492078452432 00000800:00000100:0.0:1584540130.140987:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540130.141413:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584540130.153467:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540124/real 1584540124] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540211 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584540130.153477:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584540130.153478:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584540130.153480:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584540130.153486:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540130.153488:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540130.153489:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540130.153490:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540130.153491:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540130.153493:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584540130.153494:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584540130.153496:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21930 00000100:00080000:6.0:1584540130.153499:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584540130.153513:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540130.153514:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540130.153515:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540130.153516:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540130.153517:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540130.153518:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540130.153520:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540130.153521:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540130.157522:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540130.157529:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540130.157531:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540130.157532:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540130.157534:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540130.157536:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540130.157538:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540130.157539:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540211 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584540130.157569:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078452432 new x1661492078452464 00000800:00020000:4.0:1584540136.916061:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584540136.916064:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584540136.916066:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540136.916489:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584540136.928647:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540130/real 1584540130] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540217 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584540136.928657:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584540136.928659:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584540136.928661:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584540136.928669:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540136.928671:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540136.928687:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540136.928688:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540136.928689:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540136.928691:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584540136.928692:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584540136.928694:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21936 00000100:00080000:8.0:1584540136.928697:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584540136.928703:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540136.928705:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540136.928706:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540136.928707:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540136.928708:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540136.928709:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540136.928710:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540136.928711:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540136.932730:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584540136.932738:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584540136.932740:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584540136.932741:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584540136.932743:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584540136.932746:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584540136.932747:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584540136.932749:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540217 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584540136.932759:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078452464 new x1661492078452528 00000800:00000100:0.0:1584540142.140675:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540142.141145:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540142.153219:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540136/real 1584540136] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540223 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540142.153228:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540142.153230:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540142.153231:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540142.153238:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540142.153240:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540142.153241:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540142.153242:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540142.153243:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540142.153244:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540142.153245:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540142.153247:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21942 00000100:00080000:10.0:1584540142.153249:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584540142.153260:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540142.153261:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540142.153263:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540142.153265:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540142.153265:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540142.153267:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540142.153267:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540142.153269:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540142.157272:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584540142.157277:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584540142.157278:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584540142.157279:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584540142.157280:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584540142.157282:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584540142.157284:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584540142.157286:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540223 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584540142.157315:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078452528 new x1661492078452560 00000800:00000100:0.0:1584540148.140901:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540148.141345:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540148.153397:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540142/real 1584540142] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540229 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540148.153405:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540148.153406:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540148.153408:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540148.153424:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540148.153426:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540148.153427:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540148.153428:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540148.153429:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540148.153430:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540148.153431:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540148.153433:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21948 00000100:00080000:10.0:1584540148.153435:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584540148.153443:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540148.153445:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540148.153446:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540148.153447:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540148.153447:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540148.153449:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540148.153449:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540148.153451:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540148.157417:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584540148.157422:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584540148.157424:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584540148.157425:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584540148.157426:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584540148.157428:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584540148.157430:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584540148.157431:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540229 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584540148.157460:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078452560 new x1661492078452608 00000800:00000100:0.0:1584540154.140905:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540154.141307:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540154.153361:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540148/real 1584540148] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540235 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540154.153369:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540154.153370:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540154.153372:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540154.153378:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540154.153380:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540154.153380:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540154.153382:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540154.153382:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540154.153384:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540154.153384:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540154.153387:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21954 00000100:00080000:10.0:1584540154.153389:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584540154.153398:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540154.153399:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540154.153400:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540154.153401:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540154.153402:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540154.153403:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540154.153404:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540154.153405:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540154.157509:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584540154.157513:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584540154.157514:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584540154.157515:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584540154.157517:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584540154.157520:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584540154.157521:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584540154.157523:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540235 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584540154.157533:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078452608 new x1661492078452656 00000800:00020000:2.0:1584540160.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584540160.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584540160.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540160.916493:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540160.928571:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540154/real 1584540154] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540241 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540160.928579:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540160.928580:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540160.928581:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540160.928587:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540160.928588:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540160.928597:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540160.928598:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540160.928599:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540160.928601:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540160.928602:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540160.928603:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21960 00000100:00080000:10.0:1584540160.928605:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584540160.928610:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540160.928611:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540160.928612:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540160.928613:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540160.928614:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540160.928615:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540160.928616:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540160.928617:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540160.932486:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584540160.932490:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584540160.932492:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584540160.932492:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584540160.932494:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584540160.932496:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584540160.932497:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584540160.932499:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540241 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584540160.932507:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078452656 new x1661492078452704 00000800:00000100:0.0:1584540166.141037:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540166.141424:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584540166.153472:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540160/real 1584540160] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540247 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584540166.153479:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584540166.153481:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584540166.153482:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584540166.153489:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540166.153490:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540166.153491:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540166.153492:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540166.153493:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540166.153494:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584540166.153495:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584540166.153496:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21966 00000100:00080000:12.0:1584540166.153498:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584540166.153507:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540166.153509:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540166.153509:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540166.153510:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540166.153511:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540166.153512:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540166.153513:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540166.153515:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540166.157705:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584540166.157709:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584540166.157711:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584540166.157711:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584540166.157713:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584540166.157714:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584540166.157716:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584540166.157718:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540247 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584540166.157747:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078452704 new x1661492078452736 00000800:00000100:0.0:1584540172.141140:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540172.141576:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584540172.153636:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540166/real 1584540166] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540253 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584540172.153644:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584540172.153645:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584540172.153646:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584540172.153660:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540172.153661:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540172.153662:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540172.153663:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540172.153664:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540172.153665:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584540172.153666:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584540172.153668:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21972 00000100:00080000:12.0:1584540172.153670:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584540172.153675:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540172.153677:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540172.153677:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540172.153678:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540172.153679:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540172.153681:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540172.153681:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540172.153683:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540172.157808:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584540172.157812:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584540172.157814:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584540172.157814:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584540172.157816:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584540172.157819:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584540172.157820:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584540172.157822:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540253 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540172.157831:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078452736 new x1661492078452800 00000800:00000100:0.0:1584540178.141405:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540178.141812:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540178.153862:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540172/real 1584540172] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540259 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540178.153870:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540178.153871:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540178.153873:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540178.153878:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540178.153880:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540178.153881:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540178.153882:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540178.153883:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540178.153884:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540178.153885:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540178.153887:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21978 00000100:00080000:14.0:1584540178.153889:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540178.153898:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540178.153900:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540178.153900:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540178.153901:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540178.153902:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540178.153903:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540178.153904:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540178.153905:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540178.157958:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584540178.157962:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584540178.157964:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584540178.157965:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584540178.157966:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584540178.157968:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584540178.157969:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584540178.157971:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540259 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540178.157979:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078452800 new x1661492078452832 00000800:00020000:4.0:1584540184.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584540184.916081:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584540184.916083:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540184.916535:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540184.928596:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540178/real 1584540178] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540265 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540184.928607:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540184.928609:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540184.928611:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540184.928619:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540184.928621:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540184.928637:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540184.928638:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540184.928640:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540184.928642:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540184.928643:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540184.928645:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21984 00000100:00080000:14.0:1584540184.928649:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540184.928657:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540184.928659:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540184.928660:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540184.928661:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540184.928662:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540184.928663:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540184.928664:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540184.928665:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540184.932685:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584540184.932692:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584540184.932695:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584540184.932696:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584540184.932698:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584540184.932701:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584540184.932702:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584540184.932704:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540265 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540184.932733:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078452832 new x1661492078452896 00000800:00000100:0.0:1584540190.140432:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540190.140866:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540190.152924:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540184/real 1584540184] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540271 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540190.152934:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540190.152935:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540190.152936:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540190.152943:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540190.152945:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540190.152947:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540190.152948:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540190.152949:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540190.152950:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540190.152951:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540190.152953:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21990 00000100:00080000:14.0:1584540190.152955:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540190.152967:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540190.152969:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540190.152969:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540190.152970:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540190.152971:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540190.152972:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540190.152973:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540190.152974:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540190.156955:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584540190.156961:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584540190.156963:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584540190.156965:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584540190.156967:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584540190.156969:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584540190.156971:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584540190.156973:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540271 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540190.157002:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078452896 new x1661492078452928 00000800:00000100:0.0:1584540196.140431:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540196.140866:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540196.152921:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540190/real 1584540190] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540196.152931:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540196.152932:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540196.152934:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540196.152955:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540196.152957:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540196.152958:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540196.152959:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540196.152960:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540196.152962:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540196.152962:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540196.152964:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21996 00000100:00080000:14.0:1584540196.152967:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540196.152974:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540196.152976:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540196.152977:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540196.152978:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540196.152979:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540196.152980:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540196.152981:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540196.152982:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540196.157000:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584540196.157006:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584540196.157008:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584540196.157010:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584540196.157011:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584540196.157014:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584540196.157015:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584540196.157017:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540196.157050:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078452928 new x1661492078452976 00000800:00000100:0.0:1584540202.140436:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540202.140865:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540202.152918:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540196/real 1584540196] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540202.152926:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540202.152929:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540202.152930:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540202.152937:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540202.152938:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540202.152939:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540202.152940:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540202.152941:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540202.152943:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540202.152943:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540202.152945:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22002 00000100:00080000:14.0:1584540202.152948:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540202.152958:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540202.152959:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540202.152960:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540202.152961:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540202.152962:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540202.152963:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540202.152963:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540202.152965:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540202.156973:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584540202.156979:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584540202.156981:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584540202.156982:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584540202.156983:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584540202.156985:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584540202.156987:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584540202.156989:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540202.157020:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078452976 new x1661492078453024 00000800:00020000:2.0:1584540208.916073:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584540208.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584540208.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540208.916506:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540208.928562:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540202/real 1584540202] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540208.972452:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540208.972454:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540208.972467:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540208.972472:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540208.972473:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540208.972485:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540208.972486:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540208.972487:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540208.972488:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540208.972489:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540208.972491:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22008 00000100:00080000:14.0:1584540208.972505:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540208.972512:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540208.972514:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540208.972514:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540208.972516:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540208.972516:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540208.972517:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540208.972518:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540208.972520:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540208.976416:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584540208.976422:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584540208.976424:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584540208.976425:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584540208.976427:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584540208.976430:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584540208.976431:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584540208.976433:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540208.976463:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078453024 new x1661492078453072 00000800:00000100:0.0:1584540214.140711:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540214.141144:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540214.153215:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540208/real 1584540208] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540214.153224:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540214.153225:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540214.153226:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540214.153232:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540214.153234:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540214.153235:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540214.153236:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540214.153237:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540214.153238:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540214.153239:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540214.153241:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22014 00000100:00080000:14.0:1584540214.153243:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540214.153252:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540214.153253:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540214.153254:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540214.153255:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540214.153256:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540214.153257:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540214.153258:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540214.153259:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540214.157310:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584540214.157315:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584540214.157316:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584540214.157317:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584540214.157319:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584540214.157321:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584540214.157322:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584540214.157324:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540214.157353:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078453072 new x1661492078453120 00000800:00000100:0.0:1584540220.140812:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540220.141213:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540220.153263:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540214/real 1584540214] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540220.153271:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540220.153272:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540220.153273:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540220.153286:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540220.153288:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540220.153288:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540220.153290:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540220.153290:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540220.153291:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540220.153292:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540220.153294:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22020 00000100:00080000:14.0:1584540220.153296:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540220.153301:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540220.153303:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540220.153303:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540220.153304:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540220.153306:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540220.153307:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540220.153308:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540220.153309:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540220.157386:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584540220.157390:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584540220.157391:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584540220.157392:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584540220.157393:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584540220.157395:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584540220.157397:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584540220.157398:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540220.157428:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078453120 new x1661492078453168 00000800:00000100:0.0:1584540226.140778:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540226.141213:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540226.153261:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540220/real 1584540220] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540226.153269:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540226.153270:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540226.153271:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540226.153277:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540226.153279:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540226.153279:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540226.153280:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540226.153281:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540226.153283:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540226.153283:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540226.153285:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22026 00000100:00080000:14.0:1584540226.153287:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540226.153295:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540226.153296:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540226.153298:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540226.153299:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540226.153299:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540226.153301:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540226.153301:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540226.153302:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540226.157537:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584540226.157541:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584540226.157542:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584540226.157543:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584540226.157545:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584540226.157547:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584540226.157549:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584540226.157550:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540226.157580:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078453168 new x1661492078453200 00000800:00020000:0.0:1584540232.917034:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584540232.917036:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584540232.917038:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540232.917531:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540232.929588:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540226/real 1584540226] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540232.929596:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540232.954972:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540232.954973:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540232.954978:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540232.954979:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540232.955000:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540232.955001:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540232.955002:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540232.955003:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540232.955004:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540232.955017:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22032 00000100:00080000:14.0:1584540232.955019:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540232.955026:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540232.955030:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540232.955031:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540232.955032:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540232.955033:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540232.955034:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540232.955035:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540232.955036:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540232.959065:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584540232.959070:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584540232.959071:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584540232.959072:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584540232.959073:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584540232.959075:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584540232.959077:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584540232.976859:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540232.976904:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078453200 new x1661492078453264 00000800:00000100:0.0:1584540238.141036:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540238.141434:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540238.153481:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540232/real 1584540232] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540319 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540238.153489:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540238.153490:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540238.153491:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540238.153496:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540238.153498:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540238.153498:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540238.153500:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540238.153500:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540238.153502:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540238.153502:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540238.153504:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22038 00000100:00080000:14.0:1584540238.153506:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540238.153515:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540238.153516:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540238.153517:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540238.153518:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540238.153519:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540238.153520:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540238.153521:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540238.153522:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540238.157598:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584540238.157602:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584540238.157603:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584540238.157604:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584540238.157605:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584540238.157607:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584540238.157609:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584540238.157610:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540319 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540238.157639:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078453264 new x1661492078453296 00000800:00000100:0.0:1584540244.141172:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540244.141693:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540244.153770:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540238/real 1584540238] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540325 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540244.153780:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540244.153782:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540244.153784:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540244.153804:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540244.153806:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540244.153807:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540244.153808:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540244.153820:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540244.153823:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540244.153824:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540244.153826:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22044 00000100:00080000:14.0:1584540244.153840:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540244.153848:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540244.153849:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540244.153851:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540244.153852:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540244.153852:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540244.153854:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540244.153854:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540244.153867:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540244.157992:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584540244.157999:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584540244.158001:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584540244.158002:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584540244.158004:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584540244.158006:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584540244.158008:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584540244.158010:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540325 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540244.158045:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078453296 new x1661492078453360 00000800:00000100:0.0:1584540250.140728:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540250.141163:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540250.153227:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540244/real 1584540244] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540331 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540250.153236:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540250.153238:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540250.153240:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540250.153247:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540250.153249:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540250.153250:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540250.153252:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540250.153252:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540250.153254:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540250.153255:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540250.153257:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22050 00000100:00080000:14.0:1584540250.153259:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540250.153273:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540250.153274:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540250.153275:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540250.153276:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540250.153277:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540250.153278:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540250.153279:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540250.153280:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540250.157366:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584540250.157373:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584540250.157375:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584540250.157376:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584540250.157378:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584540250.157380:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584540250.157382:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584540250.157384:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540331 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540250.157413:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078453360 new x1661492078453392 00000800:00020000:4.0:1584540256.916064:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584540256.916067:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584540256.916069:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540256.916507:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540256.928577:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540250/real 1584540250] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540337 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540256.928586:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540256.928587:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540256.928589:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540256.928596:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540256.928599:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540256.928612:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540256.928613:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540256.928614:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540256.928615:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540256.928616:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540256.928618:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22056 00000100:00080000:14.0:1584540256.928621:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540256.928627:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540256.928629:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540256.928630:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540256.928631:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540256.928631:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540256.928633:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540256.928633:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540256.928635:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540256.932511:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584540256.932518:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584540256.932520:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584540256.932521:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584540256.932523:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584540256.932526:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584540256.932527:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584540256.932530:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540337 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540256.932540:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078453392 new x1661492078453456 00000800:00000100:0.0:1584540262.140720:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540262.141128:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540262.153179:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540256/real 1584540256] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540343 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540262.153187:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540262.153188:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540262.153190:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540262.153196:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540262.153198:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540262.153198:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540262.153200:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540262.153200:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540262.153202:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540262.153202:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540262.153204:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22062 00000100:00080000:14.0:1584540262.153206:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540262.153215:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540262.153216:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540262.153217:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540262.153218:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540262.153219:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540262.153220:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540262.153220:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540262.153222:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540262.157096:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584540262.157100:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584540262.157102:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584540262.157103:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584540262.157104:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584540262.157106:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584540262.157107:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584540262.157109:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540343 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540262.157138:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078453456 new x1661492078453488 00000800:00000100:0.0:1584540268.140751:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540268.141157:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540268.153207:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540262/real 1584540262] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540349 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540268.153219:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540268.153220:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540268.153222:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540268.153235:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540268.153237:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540268.153238:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540268.153239:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540268.153239:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540268.153241:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540268.153242:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540268.153243:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22068 00000100:00080000:14.0:1584540268.153245:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540268.153251:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540268.153252:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540268.153253:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540268.153254:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540268.153254:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540268.153255:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540268.153256:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540268.153257:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540268.157400:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584540268.157404:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584540268.157406:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584540268.157407:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584540268.157408:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584540268.157410:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584540268.157411:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584540268.157413:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540349 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540268.157442:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078453488 new x1661492078453536 00000800:00000100:0.0:1584540274.140875:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540274.141337:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540274.153387:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540268/real 1584540268] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540355 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540274.153395:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540274.153396:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540274.153397:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540274.153403:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540274.153404:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540274.153405:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540274.153406:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540274.153407:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540274.153408:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540274.153409:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540274.153410:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22074 00000100:00080000:14.0:1584540274.153412:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540274.153420:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540274.153422:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540274.153422:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540274.153423:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540274.153424:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540274.153425:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540274.153427:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540274.153428:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540274.157370:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584540274.157374:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584540274.157376:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584540274.157377:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584540274.157378:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584540274.157380:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584540274.157382:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584540274.157383:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540355 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540274.157412:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078453536 new x1661492078453584 00000800:00020000:6.0:1584540280.916043:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584540280.916046:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584540280.916048:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540280.916581:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540280.928634:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540274/real 1584540274] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540361 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540280.928641:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540280.928642:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540280.928644:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540280.928649:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540280.928650:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540280.928659:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540280.928660:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540280.928661:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540280.928662:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540280.928663:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540280.928664:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22080 00000100:00080000:14.0:1584540280.928666:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540280.928672:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540280.928673:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540280.928674:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540280.928675:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540280.928676:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540280.928678:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540280.928678:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540280.928679:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540280.932641:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584540280.932645:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584540280.932647:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584540280.932648:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584540280.932649:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584540280.932651:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584540280.932652:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584540280.932654:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540361 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540280.932663:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078453584 new x1661492078453632 00000800:00000100:0.0:1584540286.141303:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540286.141834:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540286.153882:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540280/real 1584540280] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540367 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540286.153890:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540286.153891:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540286.153892:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540286.153897:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540286.153899:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540286.153899:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540286.153900:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540286.153901:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540286.153902:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540286.153903:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540286.153904:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22086 00000100:00080000:14.0:1584540286.153906:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540286.153914:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540286.153916:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540286.153917:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540286.153919:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540286.153919:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540286.153920:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540286.153921:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540286.153922:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540286.157880:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584540286.157884:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584540286.157885:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584540286.157886:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584540286.157887:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584540286.157889:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584540286.157891:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584540286.157892:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540367 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540286.157921:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078453632 new x1661492078453664 00000800:00000100:0.0:1584540292.141075:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540292.141540:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540292.153588:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540286/real 1584540286] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540373 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540292.153596:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540292.153597:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540292.153599:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540292.153612:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540292.153614:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540292.153615:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540292.153616:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540292.153617:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540292.153618:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540292.153619:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540292.153620:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22092 00000100:00080000:14.0:1584540292.153623:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540292.153629:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540292.153630:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540292.153631:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540292.153632:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540292.153633:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540292.153634:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540292.153634:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540292.153636:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540292.157478:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584540292.157483:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584540292.157484:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584540292.157485:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584540292.157486:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584540292.157488:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584540292.157489:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584540292.157491:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540373 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540292.157526:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078453664 new x1661492078453728 00000800:00000100:0.0:1584540298.141171:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540298.141632:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540298.153683:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540292/real 1584540292] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540379 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540298.153691:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540298.153692:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540298.153693:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540298.153698:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540298.153700:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540298.153701:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540298.153702:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540298.153703:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540298.153704:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540298.153705:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540298.153707:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22098 00000100:00080000:14.0:1584540298.153709:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540298.153719:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540298.153720:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540298.153721:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540298.153722:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540298.153723:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540298.153724:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540298.153724:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540298.153725:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540298.157815:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584540298.157820:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584540298.157821:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584540298.157822:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584540298.157823:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584540298.157825:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584540298.157827:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584540298.157829:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540379 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540298.157857:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078453728 new x1661492078453760 00000800:00020000:2.0:1584540304.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584540304.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584540304.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540304.916468:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540304.928522:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540298/real 1584540298] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540385 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540304.928529:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540304.928531:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540304.928532:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540304.928538:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540304.928540:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540304.928550:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540304.928551:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540304.928552:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540304.928554:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540304.928555:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540304.928557:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22104 00000100:00080000:14.0:1584540304.928560:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540304.928566:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540304.928567:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540304.928568:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540304.928569:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540304.928570:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540304.928571:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540304.928572:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540304.928573:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540304.932473:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584540304.932478:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584540304.932480:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584540304.932481:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584540304.932482:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584540304.932485:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584540304.932486:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584540304.932488:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540385 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584540304.932496:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078453760 new x1661492078453824 00000800:00000100:0.0:1584540310.142347:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540310.142927:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584540310.154985:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540304/real 1584540304] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540391 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584540310.154994:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584540310.154995:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584540310.154997:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584540310.155004:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540310.155006:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540310.155007:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540310.155008:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540310.155008:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540310.155010:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584540310.155011:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584540310.155012:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22110 00000100:00080000:16.0:1584540310.155015:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584540310.155044:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540310.155046:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540310.155046:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540310.155049:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540310.155050:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540310.155051:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540310.155052:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540310.155053:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540310.159102:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584540310.159108:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584540310.159110:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584540310.159111:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584540310.159113:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584540310.159115:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584540310.159116:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584540310.159118:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540391 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584540310.159128:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078453824 new x1661492078453856 00000800:00000100:0.0:1584540316.140492:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540316.140994:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584540316.153050:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540310/real 1584540310] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540397 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584540316.153060:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584540316.153062:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584540316.153063:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584540316.153084:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540316.153086:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540316.153087:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540316.153088:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540316.153089:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540316.153091:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584540316.153091:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584540316.153094:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22116 00000100:00080000:16.0:1584540316.153097:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584540316.153105:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540316.153108:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540316.153108:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540316.153110:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540316.153110:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540316.153112:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540316.153112:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540316.153113:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540316.157205:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584540316.157211:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584540316.157213:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584540316.157215:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584540316.157216:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584540316.157219:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584540316.157220:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584540316.157222:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540397 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584540316.157251:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078453856 new x1661492078453904 00000800:00000100:0.0:1584540322.140912:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540322.141407:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584540322.153485:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540316/real 1584540316] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540403 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584540322.153495:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584540322.153496:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584540322.153499:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584540322.153506:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540322.153508:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540322.153509:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540322.153510:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540322.153511:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540322.153512:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584540322.153513:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584540322.153515:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22122 00000100:00080000:20.0:1584540322.153518:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584540322.153533:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540322.153534:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540322.153535:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540322.153536:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540322.153537:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540322.153538:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540322.153539:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540322.153540:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540322.157877:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584540322.157884:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584540322.157886:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584540322.157887:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584540322.157889:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584540322.157891:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584540322.157893:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584540322.157895:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540403 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584540322.157924:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078453904 new x1661492078453952 00000800:00020000:2.0:1584540328.916039:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584540328.916042:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584540328.916044:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540328.916546:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584540328.928609:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540322/real 1584540322] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540409 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584540328.928623:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584540328.928625:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584540328.928626:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584540328.928634:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540328.928635:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540328.928648:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540328.928649:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540328.928650:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540328.928652:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584540328.928652:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584540328.928654:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22128 00000100:00080000:20.0:1584540328.928657:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584540328.928664:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540328.928666:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540328.928667:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540328.928668:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540328.928669:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540328.928670:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540328.928671:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540328.928672:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540328.932669:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584540328.932676:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584540328.932678:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584540328.932679:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584540328.932681:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584540328.932683:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584540328.932685:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584540328.932687:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540409 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584540328.932696:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078453952 new x1661492078454000 00000800:00000100:0.0:1584540334.140673:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540334.141170:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584540334.153220:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540328/real 1584540328] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540415 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584540334.153229:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584540334.153230:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584540334.153231:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584540334.153236:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540334.153238:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540334.153239:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540334.153240:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540334.153240:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540334.153242:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584540334.153242:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584540334.153244:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22134 00000100:00080000:20.0:1584540334.153247:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584540334.153255:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540334.153257:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540334.153257:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540334.153259:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540334.153259:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540334.153260:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540334.153262:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540334.153263:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540334.157391:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584540334.157395:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584540334.157397:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584540334.157398:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584540334.157399:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584540334.157401:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584540334.157403:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584540334.157404:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540415 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584540334.157434:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078454000 new x1661492078454048 00000800:00000100:0.0:1584540340.140700:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540340.141112:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584540340.153160:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540334/real 1584540334] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540421 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584540340.153168:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584540340.153169:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584540340.153170:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584540340.153182:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540340.153184:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540340.153184:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540340.153186:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540340.153186:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540340.153188:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584540340.153188:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584540340.153190:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22140 00000100:00080000:20.0:1584540340.153192:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584540340.153197:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540340.153198:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540340.153199:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540340.153200:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540340.153202:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540340.153203:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540340.153204:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540340.153205:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540340.157213:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584540340.157217:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584540340.157218:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584540340.157219:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584540340.157221:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584540340.157222:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584540340.157224:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584540340.157226:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540421 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584540340.157234:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078454048 new x1661492078454096 00000800:00000100:0.0:1584540346.140757:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540346.141193:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584540346.153240:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540340/real 1584540340] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540427 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584540346.153248:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584540346.153249:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584540346.153250:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584540346.153255:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540346.153256:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540346.153257:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540346.153258:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540346.153259:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540346.153260:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584540346.153261:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584540346.153263:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22146 00000100:00080000:20.0:1584540346.153265:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584540346.153273:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540346.153274:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540346.153275:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540346.153277:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540346.153277:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540346.153278:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540346.153279:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540346.153280:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540346.157254:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584540346.157258:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584540346.157259:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584540346.157260:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584540346.157261:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584540346.157263:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584540346.157265:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584540346.157266:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540427 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584540346.157296:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078454096 new x1661492078454128 00000800:00020000:2.0:1584540352.916073:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584540352.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584540352.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540352.916491:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584540352.928545:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540346/real 1584540346] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540433 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584540352.928553:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584540352.928554:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584540352.928555:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584540352.928568:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540352.928570:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540352.928580:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540352.928581:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540352.928581:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540352.928583:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584540352.928583:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584540352.928585:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22152 00000100:00080000:20.0:1584540352.928587:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584540352.928595:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540352.928596:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540352.928597:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540352.928598:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540352.928598:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540352.928600:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540352.928600:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540352.928601:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540352.932344:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584540352.932349:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584540352.932350:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584540352.932351:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584540352.932352:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584540352.932354:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584540352.932356:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584540352.932357:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540433 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584540352.932369:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078454128 new x1661492078454192 00000800:00000100:0.0:1584540358.140933:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540358.141312:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584540358.153362:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540352/real 1584540352] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540439 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584540358.153370:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584540358.153371:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584540358.153373:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584540358.153378:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540358.153379:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540358.153380:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540358.153381:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540358.153382:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540358.153383:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584540358.153384:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584540358.153386:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22158 00000100:00080000:20.0:1584540358.153388:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584540358.153397:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540358.153398:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540358.153399:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540358.153400:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540358.153400:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540358.153401:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540358.153402:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540358.153403:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584540358.157339:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584540358.157343:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584540358.157345:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584540358.157346:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584540358.157347:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584540358.157349:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584540358.157350:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584540358.157352:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540439 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584540358.157361:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078454192 new x1661492078454224 00000800:00000100:0.0:1584540364.140955:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540364.141402:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584540364.153453:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540358/real 1584540358] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540445 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584540364.153460:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584540364.153461:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584540364.153463:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584540364.153476:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540364.153478:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540364.153478:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540364.153480:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540364.153480:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540364.153482:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584540364.153483:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584540364.153485:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22164 00000100:00080000:20.0:1584540364.153487:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584540364.153493:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540364.153494:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540364.153495:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540364.153496:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540364.153496:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540364.153498:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540364.153498:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540364.153499:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540364.157517:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584540364.157521:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584540364.157523:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584540364.157524:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584540364.157526:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584540364.157528:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584540364.157529:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584540364.157531:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540445 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584540364.157539:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078454224 new x1661492078454272 00000800:00000100:0.0:1584540370.142151:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540370.142531:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584540370.154580:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540364/real 1584540364] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540451 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584540370.154588:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584540370.154589:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584540370.154591:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584540370.154595:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540370.154597:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540370.154598:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540370.154600:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540370.154600:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540370.154602:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584540370.154602:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584540370.154604:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22170 00000100:00080000:20.0:1584540370.154606:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584540370.154614:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540370.154615:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540370.154616:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540370.154617:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540370.154618:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540370.154619:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540370.154619:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540370.154620:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540370.158386:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584540370.158390:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584540370.158391:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584540370.158392:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584540370.158394:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584540370.158395:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584540370.158397:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584540370.158399:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540451 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584540370.158407:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078454272 new x1661492078454304 00000800:00020000:2.0:1584540376.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584540376.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584540376.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540376.916505:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584540376.928575:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540370/real 1584540370] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540457 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584540376.928586:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584540376.928588:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584540376.928590:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584540376.928598:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540376.928600:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540376.928614:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540376.928616:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540376.928616:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540376.928618:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584540376.928620:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584540376.928621:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22176 00000100:00080000:20.0:1584540376.928625:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584540376.928633:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540376.928634:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540376.928635:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540376.928636:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540376.928637:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540376.928638:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540376.928639:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540376.928640:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540376.932697:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584540376.932704:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584540376.932706:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584540376.932707:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584540376.932709:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584540376.932712:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584540376.932713:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584540376.932715:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540457 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584540376.932745:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078454304 new x1661492078454368 00000800:00000100:0.0:1584540382.140380:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540382.140774:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584540382.152831:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540376/real 1584540376] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540463 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584540382.152842:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584540382.152844:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584540382.152847:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584540382.152855:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540382.152857:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540382.152858:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540382.152859:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540382.152860:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540382.152862:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584540382.152863:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584540382.152865:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22182 00000100:00080000:20.0:1584540382.152868:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584540382.152886:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540382.152887:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540382.152888:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540382.152889:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540382.152890:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540382.152891:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540382.152892:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540382.152893:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540382.156905:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584540382.156912:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584540382.156916:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584540382.156917:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584540382.156919:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584540382.156922:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584540382.156923:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584540382.156925:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540463 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584540382.156934:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078454368 new x1661492078454416 00000800:00000100:0.0:1584540388.140507:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540388.140975:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584540388.153025:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540382/real 1584540382] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540469 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584540388.153042:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584540388.153043:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584540388.153044:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584540388.153060:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540388.153062:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540388.153063:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540388.153064:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540388.153064:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540388.153066:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584540388.153066:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584540388.153068:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22188 00000100:00080000:20.0:1584540388.153070:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584540388.153076:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540388.153078:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540388.153078:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540388.153080:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540388.153080:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540388.153081:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540388.153082:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540388.153083:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540388.157085:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584540388.157090:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584540388.157091:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584540388.157092:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584540388.157094:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584540388.157096:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584540388.157097:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584540388.157099:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540469 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584540388.157108:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078454416 new x1661492078454464 00000800:00000100:0.0:1584540394.140582:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540394.141089:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584540394.153142:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540388/real 1584540388] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540475 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584540394.153150:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584540394.153151:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584540394.153153:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584540394.153158:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540394.153159:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540394.153160:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540394.153161:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540394.153162:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540394.153163:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584540394.153164:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584540394.153165:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22194 00000100:00080000:20.0:1584540394.153167:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584540394.153177:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540394.153178:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540394.153179:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540394.153180:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540394.153180:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540394.153182:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540394.153183:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540394.153184:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540394.157244:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584540394.157249:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584540394.157250:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584540394.157251:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584540394.157253:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584540394.157255:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584540394.157256:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584540394.157258:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540475 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584540394.157266:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078454464 new x1661492078454512 00000800:00020000:2.0:1584540400.916070:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584540400.916072:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584540400.916074:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540400.916452:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584540400.928503:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540394/real 1584540394] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540481 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584540400.928511:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584540400.928512:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584540400.928513:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584540400.928518:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540400.928520:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540400.928528:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540400.928529:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540400.928530:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540400.928531:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584540400.928532:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584540400.928533:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22200 00000100:00080000:20.0:1584540400.928535:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584540400.928541:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540400.928542:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540400.928543:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540400.928544:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540400.928545:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540400.928546:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540400.928547:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540400.928548:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540400.932387:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584540400.932391:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584540400.932392:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584540400.932393:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584540400.932395:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584540400.932397:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584540400.932398:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584540400.932400:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540481 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584540400.932429:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078454512 new x1661492078454560 00000800:00000100:0.0:1584540406.140740:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540406.141196:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584540406.153247:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540400/real 1584540400] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540487 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584540406.153255:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584540406.153256:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584540406.153258:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584540406.153262:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540406.153264:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540406.153265:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540406.153266:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540406.153266:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540406.153268:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584540406.153268:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584540406.153270:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22206 00000100:00080000:20.0:1584540406.153272:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584540406.153290:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540406.153291:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540406.153293:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540406.153294:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540406.153294:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540406.153295:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540406.153296:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540406.153297:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540406.156994:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584540406.156997:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584540406.156999:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584540406.157000:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584540406.157001:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584540406.157003:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584540406.157004:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584540406.157006:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540487 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584540406.157036:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078454560 new x1661492078454592 00000800:00000100:0.0:1584540412.140855:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540412.141267:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584540412.153322:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540406/real 1584540406] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540493 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584540412.153330:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584540412.153331:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584540412.153332:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584540412.153347:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540412.153349:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540412.153350:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540412.153351:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540412.153351:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540412.153353:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584540412.153353:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584540412.153355:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22212 00000100:00080000:20.0:1584540412.153357:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584540412.153364:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540412.153365:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540412.153366:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540412.153367:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540412.153368:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540412.153369:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540412.153370:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540412.153371:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540412.157159:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584540412.157163:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584540412.157165:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584540412.157165:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584540412.157167:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584540412.157168:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584540412.157170:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584540412.157172:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540493 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584540412.157180:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078454592 new x1661492078454656 00000800:00000100:0.0:1584540418.140957:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540418.141419:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584540418.153469:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540412/real 1584540412] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540499 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584540418.153477:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584540418.153478:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584540418.153479:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584540418.153485:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540418.153487:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540418.153487:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540418.153488:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540418.153489:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540418.153490:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584540418.153491:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584540418.153493:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22218 00000100:00080000:20.0:1584540418.153495:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584540418.153504:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540418.153505:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540418.153506:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540418.153507:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540418.153508:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540418.153509:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540418.153509:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540418.153510:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540418.157256:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584540418.157260:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584540418.157262:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584540418.157262:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584540418.157264:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584540418.157266:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584540418.157268:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584540418.157269:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540499 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584540418.157278:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078454656 new x1661492078454688 00000800:00020000:2.0:1584540424.916073:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584540424.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584540424.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540424.916503:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584540424.928558:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540418/real 1584540418] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540505 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584540424.928567:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584540424.928568:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584540424.928570:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584540424.928575:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540424.928577:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540424.928585:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540424.928586:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540424.928587:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540424.928589:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584540424.928590:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584540424.928592:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22224 00000100:00080000:20.0:1584540424.928595:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584540424.928601:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540424.928602:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540424.928603:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540424.928604:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540424.928605:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540424.928606:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540424.928607:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540424.928608:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540424.932701:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584540424.932706:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584540424.932708:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584540424.932709:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584540424.932711:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584540424.932713:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584540424.932715:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584540424.932717:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540505 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584540424.932745:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078454688 new x1661492078454752 00000800:00000100:0.0:1584540430.141118:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540430.141615:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584540430.153663:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540424/real 1584540424] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540511 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584540430.153671:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584540430.153672:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584540430.153673:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584540430.153678:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540430.153680:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540430.153680:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540430.153682:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540430.153683:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540430.153684:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584540430.153685:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584540430.153686:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22230 00000100:00080000:20.0:1584540430.153688:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584540430.153696:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540430.153698:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540430.153698:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540430.153700:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540430.153700:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540430.153701:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540430.153702:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540430.153703:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540430.157902:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584540430.157906:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584540430.157908:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584540430.157909:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584540430.157910:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584540430.157912:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584540430.157913:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584540430.157915:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540511 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584540430.157944:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078454752 new x1661492078454784 00000800:00000100:0.0:1584540436.141246:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540436.141931:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584540436.153981:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540430/real 1584540430] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540517 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584540436.153989:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584540436.153991:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584540436.153992:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584540436.154006:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540436.154008:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540436.154009:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540436.154010:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540436.154011:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540436.154023:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584540436.154024:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584540436.154026:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22236 00000100:00080000:20.0:1584540436.154159:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584540436.154165:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540436.154167:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540436.154167:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540436.154169:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540436.154169:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540436.154170:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540436.154171:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584540436.154172:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540436.158447:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584540436.158451:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584540436.158453:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584540436.158454:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584540436.158456:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584540436.158458:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584540436.158459:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584540436.158461:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540517 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584540436.158494:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078454784 new x1661492078454832 00000800:00000100:0.0:1584540442.141256:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540442.141765:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584540442.153840:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540436/real 1584540436] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540523 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584540442.153849:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584540442.153851:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584540442.153853:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584540442.153861:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540442.153863:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540442.153864:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540442.153865:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540442.153866:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540442.153868:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584540442.153869:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584540442.153871:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22242 00000100:00080000:22.0:1584540442.153874:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584540442.153889:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540442.153891:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540442.153891:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540442.153892:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540442.153893:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540442.153894:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540442.153895:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540442.153907:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540442.158005:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584540442.158012:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584540442.158014:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584540442.158016:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584540442.158018:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584540442.158020:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584540442.158022:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584540442.158024:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540523 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584540442.158059:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078454832 new x1661492078454880 00000800:00000100:0.0:1584540448.140539:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540448.141025:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584540448.153126:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540442/real 1584540442] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540529 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584540448.153137:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584540448.153138:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584540448.153141:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584540448.153149:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540448.153151:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540448.153166:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540448.153168:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540448.153168:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540448.153170:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584540448.153171:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584540448.153185:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22248 00000100:00080000:22.0:1584540448.153188:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584540448.153195:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540448.153197:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540448.153198:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540448.153199:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540448.153200:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540448.153201:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540448.153202:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540448.153203:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540448.157502:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584540448.157513:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584540448.157517:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584540448.157519:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584540448.157522:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584540448.157527:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584540448.157531:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584540448.157535:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540529 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584540448.157553:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078454880 new x1661492078454928 00000800:00000100:0.0:1584540454.140607:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540454.141015:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584540454.153089:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540448/real 1584540448] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540535 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584540454.153097:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584540454.153099:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584540454.153100:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584540454.153106:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540454.153108:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540454.153108:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540454.153109:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540454.153110:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540454.153111:0:176857:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584540454.153112:0:176857:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584540454.153114:0:176857:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22254 00000100:00080000:2.0:1584540454.153116:0:176857:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584540454.153127:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540454.153128:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540454.153129:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540454.153130:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540454.153131:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540454.153133:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540454.153133:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540454.153146:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540454.157352:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584540454.157357:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584540454.157359:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584540454.157360:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584540454.157363:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584540454.157365:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584540454.157367:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584540454.157369:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540535 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584540454.157378:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078454928 new x1661492078454976 00000800:00000100:0.0:1584540460.140645:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540460.141077:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584540460.153145:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540454/real 1584540454] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540541 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584540460.153152:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584540460.153154:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584540460.153155:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584540460.153169:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540460.153170:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540460.153171:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540460.153172:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540460.153173:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540460.153174:0:176857:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584540460.153175:0:176857:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584540460.153176:0:176857:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22260 00000100:00080000:2.0:1584540460.153178:0:176857:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584540460.153183:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540460.153185:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540460.153185:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540460.153187:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540460.153188:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540460.153189:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540460.153190:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540460.153191:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540460.157231:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584540460.157235:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584540460.157236:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584540460.157237:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584540460.157239:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584540460.157241:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584540460.157242:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584540460.157244:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540541 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584540460.157253:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078454976 new x1661492078455024 00000800:00000100:0.0:1584540466.140829:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540466.141401:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584540466.153451:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540460/real 1584540460] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540547 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584540466.153458:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584540466.153459:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584540466.153460:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584540466.153465:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540466.153466:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540466.153467:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540466.153468:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540466.153469:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540466.153470:0:176857:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584540466.153471:0:176857:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584540466.153472:0:176857:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22266 00000100:00080000:2.0:1584540466.153474:0:176857:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584540466.153482:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540466.153484:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540466.153485:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540466.153486:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540466.153487:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540466.153488:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540466.153489:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540466.153490:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540466.157532:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584540466.157536:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584540466.157538:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584540466.157539:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584540466.157541:0:25724:0:(client.c:2816:ptlrpc_resend_req()) @@@ going to resend req@ffff8d2babe40900 x1661492078455040/t0(0) o17->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 456/432 e 0 to 0 dl 1584540538 ref 1 fl Rpc:/0/ffffffff rc 0/-1 00000100:00080000:8.0:1584540466.157547:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584540466.157549:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584540466.157550:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584540466.157552:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540547 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584540466.157560:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078455024 new x1661492078455088 00000800:00000100:0.0:1584540472.140806:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540472.141300:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584540472.153370:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540466/real 1584540466] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540553 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584540472.153377:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584540472.153379:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584540472.153380:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584540472.153386:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540472.153387:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540472.153388:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540472.153389:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540472.153390:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540472.153391:0:176857:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584540472.153392:0:176857:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584540472.153394:0:176857:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22272 00000100:00080000:2.0:1584540472.153397:0:176857:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584540472.153406:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540472.153407:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540472.153408:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540472.153409:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540472.153409:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540472.153411:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540472.153411:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540472.153412:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540472.157471:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584540472.157475:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584540472.157477:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584540472.157478:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584540472.157480:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584540472.157481:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584540472.157483:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584540472.157485:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540553 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584540472.157494:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078455088 new x1661492078455136 00000800:00000100:0.0:1584540478.140868:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540478.141393:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584540478.153472:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540472/real 1584540472] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540559 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584540478.153485:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584540478.153486:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584540478.153487:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584540478.153493:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540478.153495:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540478.153496:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540478.153497:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540478.153497:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540478.153499:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584540478.153499:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584540478.153501:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22278 00000100:00080000:6.0:1584540478.153503:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584540478.153512:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540478.153513:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540478.153514:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540478.153516:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540478.153518:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540478.153520:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540478.153520:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540478.153522:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540478.157516:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584540478.157520:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584540478.157522:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584540478.157522:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584540478.157524:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584540478.157525:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584540478.157527:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584540478.157529:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540559 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584540478.157558:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078455136 new x1661492078455168 00000800:00000100:0.0:1584540484.141353:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540484.141865:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584540484.153944:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540478/real 1584540478] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540565 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584540484.153955:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584540484.153957:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584540484.153959:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584540484.153980:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540484.153982:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540484.153983:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540484.153984:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540484.153985:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540484.153997:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584540484.153998:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584540484.154001:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22284 00000100:00080000:6.0:1584540484.154004:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584540484.154023:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540484.154024:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540484.154025:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540484.154046:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540484.154046:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540484.154048:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540484.154049:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540484.154050:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540484.158263:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584540484.158270:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584540484.158273:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584540484.158274:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584540484.158276:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584540484.158279:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584540484.158280:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584540484.158282:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540565 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584540484.158292:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078455168 new x1661492078455232 00000800:00000100:0.0:1584540490.140699:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540490.141100:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584540490.153171:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540484/real 1584540484] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540571 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584540490.153180:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584540490.153182:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584540490.153184:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584540490.153191:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540490.153193:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540490.153206:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540490.153207:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540490.153208:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540490.153209:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584540490.153210:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584540490.153212:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22290 00000100:00080000:6.0:1584540490.153215:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584540490.153222:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540490.153223:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540490.153224:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540490.153226:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540490.153227:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540490.153228:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540490.153229:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540490.153230:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540490.157214:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584540490.157221:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584540490.157223:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584540490.157224:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584540490.157226:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584540490.157229:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584540490.157230:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584540490.157232:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540571 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584540490.157243:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078455232 new x1661492078455280 00000800:00000100:0.0:1584540496.140695:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540496.141199:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584540496.153269:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540490/real 1584540490] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540577 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584540496.153278:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584540496.153280:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584540496.153281:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584540496.153288:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540496.153290:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540496.153290:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540496.153291:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540496.153292:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540496.153294:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584540496.153295:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584540496.153296:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22296 00000100:00080000:6.0:1584540496.153299:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584540496.153311:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540496.153313:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540496.153315:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540496.153316:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540496.153317:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540496.153318:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540496.153319:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540496.153320:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540496.157193:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584540496.157199:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584540496.157201:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584540496.157202:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584540496.157204:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584540496.157206:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584540496.157208:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584540496.157209:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540577 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584540496.157238:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078455280 new x1661492078455312 00000800:00000100:0.0:1584540502.140751:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540502.141269:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584540502.153340:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540496/real 1584540496] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540583 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584540502.153348:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584540502.153349:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584540502.153350:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584540502.153356:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540502.153358:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540502.153358:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540502.153360:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540502.153360:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540502.153362:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584540502.153362:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584540502.153364:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22302 00000100:00080000:6.0:1584540502.153366:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584540502.153377:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540502.153378:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540502.153379:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540502.153381:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540502.153381:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540502.153382:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540502.153383:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540502.153384:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540502.157290:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584540502.157295:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584540502.157297:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584540502.157298:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584540502.157299:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584540502.157302:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584540502.157303:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584540502.157305:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540583 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584540502.157334:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078455312 new x1661492078455360 00000800:00000100:0.0:1584540508.140894:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540508.141413:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584540508.153472:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540502/real 1584540502] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540589 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584540508.153479:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584540508.153480:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584540508.153482:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584540508.153496:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540508.153498:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540508.153498:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540508.153499:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540508.153500:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540508.153501:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584540508.153502:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584540508.153505:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22308 00000100:00080000:6.0:1584540508.153507:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584540508.153512:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540508.153513:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540508.153514:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540508.153515:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540508.153515:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540508.153516:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540508.153517:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540508.153518:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540508.157406:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584540508.157410:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584540508.157411:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584540508.157412:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584540508.157414:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584540508.157416:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584540508.157417:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584540508.157419:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540589 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584540508.157447:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078455360 new x1661492078455408 00000800:00000100:0.0:1584540514.140861:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540514.141292:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584540514.153383:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540508/real 1584540508] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540595 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584540514.153391:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584540514.153392:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584540514.153394:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584540514.153400:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540514.153401:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540514.153412:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540514.153413:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540514.153414:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540514.153416:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584540514.153417:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584540514.153419:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22314 00000100:00080000:6.0:1584540514.153421:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584540514.153426:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540514.153428:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540514.153428:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540514.153429:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540514.153430:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540514.153431:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540514.153432:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540514.153433:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540514.157534:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584540514.157538:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584540514.157540:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584540514.157541:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584540514.157542:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584540514.157544:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584540514.157546:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584540514.157547:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540595 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584540514.157576:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078455408 new x1661492078455472 00000800:00000100:0.0:1584540520.141931:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540520.142461:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584540520.154530:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540514/real 1584540514] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540601 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584540520.154538:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584540520.154539:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584540520.154540:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584540520.154546:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540520.154547:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540520.154548:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540520.154550:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540520.154551:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540520.154552:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584540520.154553:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584540520.154554:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22320 00000100:00080000:6.0:1584540520.154556:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584540520.154565:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540520.154566:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540520.154567:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540520.154568:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540520.154568:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540520.154569:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540520.154570:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540520.154571:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540520.158405:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584540520.158409:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584540520.158411:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584540520.158412:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584540520.158414:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584540520.158416:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584540520.158417:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584540520.158419:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540601 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584540520.158428:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078455472 new x1661492078455504 00000800:00000100:0.0:1584540526.141063:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540526.141592:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584540526.153645:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540520/real 1584540520] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540607 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584540526.153655:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584540526.153656:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584540526.153658:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584540526.153664:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540526.153667:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540526.153668:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540526.153669:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540526.153670:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540526.153671:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584540526.153672:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584540526.153674:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22326 00000100:00080000:6.0:1584540526.153677:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584540526.153690:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540526.153691:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540526.153692:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540526.153693:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540526.153694:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540526.153695:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540526.153696:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540526.153697:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540526.157663:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584540526.157670:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584540526.157672:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584540526.157673:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584540526.157675:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584540526.157677:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584540526.157679:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584540526.157681:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540607 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584540526.157689:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078455504 new x1661492078455536 00000800:00000100:0.0:1584540532.140963:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540532.141440:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584540532.153512:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540526/real 1584540526] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540613 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584540532.153521:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584540532.153523:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584540532.153525:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584540532.153543:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540532.153545:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540532.153545:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540532.153547:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540532.153547:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540532.153549:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584540532.153550:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584540532.153552:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22332 00000100:00080000:6.0:1584540532.153555:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584540532.153562:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540532.153564:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540532.153564:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540532.153566:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540532.153566:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540532.153568:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540532.153568:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540532.153569:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540532.157488:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584540532.157494:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584540532.157496:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584540532.157496:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584540532.157498:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584540532.157501:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584540532.157502:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584540532.157504:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540613 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584540532.157533:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078455536 new x1661492078455600 00000800:00000100:0.0:1584540538.141127:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540538.141565:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584540538.153616:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540532/real 1584540532] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540619 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584540538.153628:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584540538.153629:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584540538.153630:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584540538.153635:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540538.153637:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540538.153647:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540538.153649:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540538.153649:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540538.153651:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584540538.153652:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584540538.153653:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22338 00000100:00080000:6.0:1584540538.153655:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584540538.153661:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540538.153663:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540538.153663:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540538.153664:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540538.153665:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540538.153666:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540538.153667:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540538.153668:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00000400:14.0:1584540538.154041:0:25732:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584540465/real 1584540465] req@ffff8d2babe43a80 x1661492078455056/t0(0) o17->public1-OST0001-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 456/432 e 0 to 1 dl 1584540538 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 00000100:02000400:14.0:1584540538.154053:0:25732:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection to public1-OST0001 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540538.154055:0:25732:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540538.154056:0:25732:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0001_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540538.154061:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540538.154063:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540538.154063:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540538.154065:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:14.0:1584540538.154065:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540538.154067:0:4057:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 21245 00000100:00080000:14.0:1584540538.154069:0:4057:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540538.154078:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540538.154079:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540538.154080:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540538.154081:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540538.154082:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540538.154083:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540538.154084:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540538.154085:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540538.154086:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540538.154087:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540538.157548:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584540538.157554:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584540538.157555:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0001_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584540538.157556:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584540538.157557:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584540538.157560:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584540538.157561:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584540538.157563:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727c00): req@ffff8d2babe43a80 x1661492078455056/t0(0) o17->public1-OST0001-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 456/432 e 0 to 1 dl 1584540538 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 00000100:00080000:16.0:1584540538.157571:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584540538.157573:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584540538.157573:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584540538.157574:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584540538.157575:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584540538.157576:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584540538.157577:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584540538.157579:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540619 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584540538.157586:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078455600 new x1661492078455664 00000800:00000100:0.0:1584540544.141268:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540544.141674:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584540544.153730:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540538/real 1584540538] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540625 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584540544.153740:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584540544.153742:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584540544.153743:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584540544.153749:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540544.153751:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540544.153752:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540544.153754:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540544.153754:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540544.153756:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584540544.153757:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584540544.153759:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22344 00000100:00080000:6.0:1584540544.153762:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584540544.153775:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540544.153776:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540544.153777:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540544.153778:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540544.153779:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540544.153780:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540544.153782:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540544.153783:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540544.157501:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584540544.157509:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584540544.157511:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584540544.157512:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584540544.157514:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584540544.157516:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584540544.157518:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584540544.157520:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540625 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584540544.157549:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078455664 new x1661492078455712 00000800:00020000:2.0:1584540550.916056:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584540550.938872:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584540550.963046:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540550.963469:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584540550.975529:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540544/real 1584540544] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540631 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584540550.975540:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584540550.975542:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584540550.975544:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584540550.975553:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540550.975555:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540550.975556:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540550.975557:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540550.975559:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540550.975561:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584540550.975562:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584540550.975564:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22350 00000100:00080000:8.0:1584540550.975568:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584540550.975585:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540550.975587:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540550.975587:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540550.975589:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540550.975589:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540550.975591:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540550.975591:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540550.975593:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540550.979471:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584540550.979478:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584540550.979481:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584540550.979482:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584540550.979484:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584540550.979487:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584540550.979488:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584540550.979490:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540631 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584540550.979498:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078455712 new x1661492078455744 00000800:00000100:0.0:1584540556.140539:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540556.141072:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584540556.153120:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540550/real 1584540550] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540637 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584540556.153128:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584540556.153129:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584540556.153130:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584540556.153144:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540556.153146:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540556.153148:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540556.153149:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540556.153150:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540556.153151:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584540556.153152:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584540556.153153:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22356 00000100:00080000:8.0:1584540556.153155:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584540556.153161:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540556.153163:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540556.153163:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540556.153164:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540556.153165:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540556.153166:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540556.153167:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540556.153168:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540556.157008:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584540556.157012:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584540556.157014:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584540556.157014:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584540556.157016:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584540556.157018:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584540556.157019:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584540556.157021:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540637 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584540556.157051:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078455744 new x1661492078455792 00000800:00000100:0.0:1584540562.140571:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540562.141006:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584540562.153058:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540556/real 1584540556] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540643 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584540562.153066:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584540562.153067:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584540562.153068:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584540562.153075:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540562.153077:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540562.153086:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540562.153087:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540562.153088:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540562.153089:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584540562.153090:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584540562.153092:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22362 00000100:00080000:8.0:1584540562.153094:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584540562.153099:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540562.153101:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540562.153101:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540562.153102:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540562.153103:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540562.153104:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540562.153105:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540562.153106:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540562.156895:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584540562.156900:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584540562.156901:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584540562.156902:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584540562.156903:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584540562.156906:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584540562.156907:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584540562.156909:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540643 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584540562.156918:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078455792 new x1661492078455856 00000800:00000100:0.0:1584540568.140640:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540568.141058:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540568.153108:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540562/real 1584540562] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540649 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540568.153116:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540568.153117:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540568.153118:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540568.153125:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540568.153126:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540568.153127:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540568.153129:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540568.153130:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540568.153131:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540568.153132:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540568.153134:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22368 00000100:00080000:10.0:1584540568.153136:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584540568.153145:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540568.153146:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540568.153147:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540568.153148:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540568.153148:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540568.153150:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540568.153150:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540568.153152:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540568.156960:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584540568.156964:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584540568.156966:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584540568.156966:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584540568.156968:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584540568.156970:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584540568.156971:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584540568.156973:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540649 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584540568.156982:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078455856 new x1661492078455888 00000800:00020000:4.0:1584540574.916057:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584540574.916060:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584540574.916062:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540574.916473:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540574.928550:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540568/real 1584540568] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540655 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540574.928557:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540574.928558:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540574.928560:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540574.928565:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540574.928568:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540574.928569:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540574.928570:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540574.928570:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540574.928572:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540574.928572:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540574.928574:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22374 00000100:00080000:10.0:1584540574.928576:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584540574.928584:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540574.928586:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540574.928586:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540574.928587:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540574.928588:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540574.928589:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540574.928590:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540574.928591:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540574.932427:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584540574.932432:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584540574.932433:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584540574.932434:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584540574.932435:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584540574.932437:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584540574.932439:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584540574.932441:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540655 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584540574.932450:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078455888 new x1661492078455936 00000800:00000100:0.0:1584540580.141791:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540580.142354:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540580.154407:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540574/real 1584540574] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540661 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540580.154415:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540580.154417:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540580.154420:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540580.154435:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540580.154437:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540580.154438:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540580.154439:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540580.154440:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540580.154442:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540580.154442:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540580.154444:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22380 00000100:00080000:10.0:1584540580.154447:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584540580.154454:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540580.154455:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540580.154456:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540580.154457:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540580.154458:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540580.154459:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540580.154459:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540580.154460:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540580.158345:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584540580.158351:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584540580.158353:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584540580.158354:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584540580.158356:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584540580.158357:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584540580.158359:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584540580.158361:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540661 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584540580.158393:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078455936 new x1661492078456000 00000800:00000100:0.0:1584540586.140867:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540586.141376:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584540586.153425:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540580/real 1584540580] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540667 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584540586.153433:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584540586.153434:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584540586.153436:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584540586.153442:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540586.153444:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540586.153453:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540586.153454:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540586.153456:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540586.153457:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584540586.153458:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584540586.153460:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22386 00000100:00080000:12.0:1584540586.153462:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584540586.153467:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540586.153468:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540586.153469:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540586.153470:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540586.153471:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540586.153472:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540586.153472:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540586.153474:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540586.157326:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584540586.157330:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584540586.157331:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584540586.157332:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584540586.157333:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584540586.157336:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584540586.157337:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584540586.157339:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540667 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584540586.157367:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078456000 new x1661492078456048 00000800:00000100:0.0:1584540592.140996:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540592.141423:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584540592.153476:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540586/real 1584540586] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540673 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584540592.153483:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584540592.153484:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584540592.153486:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584540592.153492:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540592.153494:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540592.153495:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540592.153496:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540592.153497:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540592.153498:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584540592.153499:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584540592.153501:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22392 00000100:00080000:12.0:1584540592.153503:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584540592.153511:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540592.153512:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540592.153513:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540592.153514:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540592.153515:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540592.153516:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540592.153516:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540592.153517:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540592.157217:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584540592.157221:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584540592.157223:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584540592.157224:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584540592.157225:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584540592.157227:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584540592.157228:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584540592.157230:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540673 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584540592.157239:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078456048 new x1661492078456096 00000800:00020000:4.0:1584540598.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584540598.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584540598.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540598.916531:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584540598.928618:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540592/real 1584540592] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540679 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584540598.928636:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584540598.928637:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584540598.928639:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584540598.928646:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540598.928647:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540598.928648:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540598.928649:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540598.928650:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540598.928651:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584540598.928652:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584540598.928654:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22398 00000100:00080000:12.0:1584540598.928656:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584540598.928664:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540598.928665:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540598.928666:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540598.928667:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540598.928667:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540598.928668:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540598.928669:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540598.928670:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540598.932567:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584540598.932571:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584540598.932573:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584540598.932573:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584540598.932575:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584540598.932576:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584540598.932578:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584540598.932580:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540679 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584540598.932610:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078456096 new x1661492078456128 00000800:00000100:0.0:1584540604.142152:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540604.142577:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584540604.154632:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540598/real 1584540598] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540685 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584540604.154643:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584540604.154645:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584540604.154647:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584540604.154668:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540604.154670:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540604.154671:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540604.154673:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540604.154674:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540604.154675:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584540604.154677:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584540604.154679:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22404 00000100:00080000:12.0:1584540604.154682:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584540604.154690:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540604.154691:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540604.154692:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540604.154693:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540604.154694:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540604.154696:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540604.154696:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540604.154698:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540604.158800:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584540604.158808:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584540604.158810:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584540604.158811:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584540604.158813:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584540604.158816:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584540604.158817:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584540604.158819:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540685 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584540604.158828:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078456128 new x1661492078456192 00000800:00000100:0.0:1584540610.141290:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540610.141737:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584540610.153794:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540604/real 1584540604] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540691 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584540610.153803:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584540610.153804:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584540610.153806:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584540610.153812:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540610.153814:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540610.153825:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540610.153826:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540610.153827:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540610.153829:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584540610.153830:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584540610.153831:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22410 00000100:00080000:12.0:1584540610.153834:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584540610.153840:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540610.153841:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540610.153842:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540610.153843:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540610.153844:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540610.153845:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540610.153846:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540610.153847:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540610.157737:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584540610.157742:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584540610.157744:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584540610.157745:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584540610.157747:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584540610.157749:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584540610.157751:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584540610.157752:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540691 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584540610.157761:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078456192 new x1661492078456240 00000800:00000100:0.0:1584540616.140481:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540616.141032:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584540616.153088:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540610/real 1584540610] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540697 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584540616.153099:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584540616.153100:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584540616.153102:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584540616.153111:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540616.153113:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540616.153114:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540616.153115:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540616.153116:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540616.153118:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584540616.153119:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584540616.153121:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22416 00000100:00080000:12.0:1584540616.153123:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584540616.153139:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540616.153141:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540616.153141:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540616.153143:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540616.153143:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540616.153145:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540616.153147:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540616.153148:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540616.157242:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584540616.157249:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584540616.157251:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584540616.157252:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584540616.157254:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584540616.157256:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584540616.157258:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584540616.157260:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540697 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584540616.157270:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078456240 new x1661492078456272 00000800:00020000:4.0:1584540622.916070:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584540622.916073:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584540622.916075:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540622.916486:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584540622.928578:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540616/real 1584540616] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540703 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584540622.928597:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584540622.928599:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584540622.928600:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584540622.928606:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540622.928608:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540622.928609:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540622.928610:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540622.928611:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540622.928612:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584540622.928613:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584540622.928615:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22422 00000100:00080000:12.0:1584540622.928617:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584540622.928630:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540622.928631:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540622.928632:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540622.928633:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540622.928635:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540622.928636:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540622.928648:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540622.928649:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540622.932862:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584540622.932878:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584540622.932880:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584540622.932881:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584540622.932882:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584540622.932885:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584540622.932886:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584540622.932888:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540703 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584540622.932919:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078456272 new x1661492078456320 00000800:00000100:0.0:1584540628.140695:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540628.141139:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584540628.153193:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540622/real 1584540622] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540709 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584540628.153201:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584540628.153202:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584540628.153204:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584540628.153222:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540628.153224:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540628.153225:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540628.153226:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540628.153227:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540628.153228:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584540628.153229:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584540628.153231:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22428 00000100:00080000:12.0:1584540628.153233:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584540628.153240:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540628.153241:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540628.153244:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540628.153245:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540628.153246:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540628.153247:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540628.153247:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540628.153249:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540628.157136:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584540628.157141:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584540628.157142:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584540628.157143:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584540628.157144:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584540628.157146:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584540628.157148:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584540628.157150:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540709 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584540628.157158:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078456320 new x1661492078456368 00000800:00000100:0.0:1584540634.140833:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540634.141295:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584540634.153355:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540628/real 1584540628] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540715 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584540634.153363:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584540634.153364:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584540634.153366:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584540634.153371:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540634.153373:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540634.153383:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540634.153384:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540634.153385:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540634.153386:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584540634.153387:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584540634.153389:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22434 00000100:00080000:12.0:1584540634.153391:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584540634.153398:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540634.153399:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540634.153400:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540634.153401:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540634.153402:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540634.153403:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540634.153403:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540634.153405:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540634.157263:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584540634.157268:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584540634.157269:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584540634.157270:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584540634.157271:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584540634.157273:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584540634.157275:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584540634.157277:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540715 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584540634.157286:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078456368 new x1661492078456432 00000800:00000100:0.0:1584540640.140711:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540640.141148:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584540640.153197:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540634/real 1584540634] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540721 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584540640.153205:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584540640.153206:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584540640.153207:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584540640.153213:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540640.153214:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540640.153215:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540640.153216:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540640.153217:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540640.153218:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584540640.153219:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584540640.153221:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22440 00000100:00080000:12.0:1584540640.153223:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584540640.153231:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540640.153232:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540640.153233:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540640.153234:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540640.153235:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540640.153236:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540640.153236:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540640.153238:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540640.157234:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584540640.157238:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584540640.157240:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584540640.157241:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584540640.157242:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584540640.157244:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584540640.157245:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584540640.157247:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540721 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584540640.157256:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078456432 new x1661492078456464 00000800:00020000:4.0:1584540646.916119:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584540646.916125:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584540646.916129:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540646.916582:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584540646.928692:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540640/real 1584540640] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540727 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584540646.928700:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584540646.928701:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584540646.928702:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584540646.928708:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540646.928710:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540646.928710:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540646.928712:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540646.928712:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540646.928714:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584540646.928715:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584540646.928717:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22446 00000100:00080000:12.0:1584540646.928719:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584540646.928727:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540646.928729:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540646.928729:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540646.928730:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540646.928731:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540646.928732:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540646.928733:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540646.928734:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540646.932469:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584540646.932474:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584540646.932475:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584540646.932476:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584540646.932477:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584540646.932480:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584540646.932481:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584540646.932483:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540727 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584540646.932492:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078456464 new x1661492078456512 00000800:00000100:0.0:1584540652.140925:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540652.141358:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584540652.153429:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540646/real 1584540646] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540733 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584540652.153437:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584540652.153438:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584540652.153439:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584540652.153454:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540652.153455:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540652.153456:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540652.153458:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540652.153459:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540652.153460:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584540652.153472:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584540652.153473:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22452 00000100:00080000:12.0:1584540652.153475:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584540652.153481:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540652.153482:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540652.153483:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540652.153484:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540652.153484:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540652.153486:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540652.153486:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540652.153487:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540652.157410:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584540652.157414:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584540652.157415:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584540652.157416:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584540652.157417:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584540652.157419:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584540652.157421:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584540652.157422:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540733 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584540652.157456:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078456512 new x1661492078456560 00000800:00000100:0.0:1584540658.141123:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540658.141555:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584540658.153606:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540652/real 1584540652] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540739 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584540658.153614:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584540658.153615:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584540658.153616:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584540658.153622:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540658.153625:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540658.153635:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540658.153636:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540658.153637:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540658.153638:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584540658.153638:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584540658.153640:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22458 00000100:00080000:12.0:1584540658.153653:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584540658.153658:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540658.153660:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540658.153660:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540658.153661:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540658.153662:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540658.153663:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540658.153664:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540658.153665:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540658.157465:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584540658.157469:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584540658.157470:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584540658.157471:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584540658.157473:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584540658.157475:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584540658.157477:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584540658.157478:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540739 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584540658.157508:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078456560 new x1661492078456608 00000800:00000100:0.0:1584540664.141222:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540664.141727:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584540664.153783:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540658/real 1584540658] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540745 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584540664.153793:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584540664.153795:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584540664.153798:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584540664.153806:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540664.153808:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540664.153808:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540664.153810:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540664.153810:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540664.153812:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584540664.153813:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584540664.153815:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22464 00000100:00080000:12.0:1584540664.153819:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584540664.153833:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540664.153834:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540664.153835:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540664.153836:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540664.153837:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540664.153838:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540664.153839:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540664.153840:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540664.157977:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584540664.157983:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584540664.157986:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584540664.157987:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584540664.157989:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584540664.157992:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584540664.157994:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584540664.157996:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540745 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584540664.158025:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078456608 new x1661492078456656 00000800:00020000:2.0:1584540670.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584540670.916086:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584540670.916091:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540670.916524:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584540670.928611:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540664/real 1584540664] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540751 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584540670.928636:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584540670.928638:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584540670.928640:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584540670.928648:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540670.928650:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540670.928651:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540670.928652:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540670.928653:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540670.928654:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584540670.928655:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584540670.928657:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22470 00000100:00080000:12.0:1584540670.928660:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584540670.928675:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540670.928676:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540670.928677:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540670.928678:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540670.928678:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540670.928680:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540670.928680:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540670.928693:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540670.932845:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584540670.932861:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584540670.932864:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584540670.932865:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584540670.932867:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584540670.932869:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584540670.932870:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584540670.932872:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540751 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584540670.932904:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078456656 new x1661492078456688 00000800:00000100:0.0:1584540676.140632:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540676.141167:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584540676.153221:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540670/real 1584540670] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540757 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584540676.153229:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584540676.153231:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584540676.153232:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584540676.153253:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540676.153255:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540676.153256:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540676.153257:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540676.153258:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540676.153259:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584540676.153260:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584540676.153262:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22476 00000100:00080000:12.0:1584540676.153265:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584540676.153272:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540676.153274:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540676.153275:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540676.153276:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540676.153276:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540676.153278:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540676.153279:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540676.153281:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540676.157160:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584540676.157166:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584540676.157168:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584540676.157169:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584540676.157172:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584540676.157175:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584540676.157176:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584540676.157178:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540757 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584540676.157208:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078456688 new x1661492078456736 00000800:00000100:0.0:1584540682.140721:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540682.141153:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584540682.153205:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540676/real 1584540676] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540763 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584540682.153214:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584540682.153215:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584540682.153216:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584540682.153222:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540682.153224:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540682.153234:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540682.153235:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540682.153236:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540682.153237:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584540682.153238:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584540682.153240:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22482 00000100:00080000:12.0:1584540682.153242:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584540682.153249:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540682.153250:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540682.153251:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540682.153252:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540682.153254:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540682.153255:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540682.153256:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540682.153257:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540682.157189:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584540682.157195:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584540682.157196:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584540682.157198:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584540682.157199:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584540682.157202:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584540682.157203:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584540682.157205:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540763 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584540682.157234:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078456736 new x1661492078456800 00000800:00000100:0.0:1584540688.140769:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540688.141236:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584540688.153304:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540682/real 1584540682] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540769 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584540688.153311:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584540688.153312:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584540688.153314:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584540688.153319:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540688.153321:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540688.153322:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540688.153323:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540688.153324:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540688.153325:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584540688.153326:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584540688.153327:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22488 00000100:00080000:12.0:1584540688.153329:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584540688.153337:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540688.153338:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540688.153340:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540688.153341:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540688.153342:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540688.153343:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540688.153343:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540688.153356:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540688.157294:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584540688.157298:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584540688.157300:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584540688.157301:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584540688.157302:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584540688.157304:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584540688.157305:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584540688.157307:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540769 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584540688.157336:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078456800 new x1661492078456832 00000800:00020000:2.0:1584540694.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584540694.916081:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584540694.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540694.916642:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584540694.928735:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540688/real 1584540688] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540775 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584540694.928743:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584540694.928744:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584540694.928746:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584540694.928763:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540694.928765:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540694.928766:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540694.928767:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540694.928768:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540694.928769:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584540694.928770:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584540694.928771:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22494 00000100:00080000:12.0:1584540694.928774:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584540694.928784:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540694.928785:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540694.928786:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540694.928787:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540694.928788:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540694.928789:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540694.928789:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540694.928791:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540694.932823:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584540694.932828:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584540694.932830:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584540694.932831:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584540694.932833:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584540694.932835:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584540694.932836:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584540694.932838:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540775 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584540694.932868:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078456832 new x1661492078456880 00000800:00000100:0.0:1584540700.140836:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540700.141273:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584540700.153322:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540694/real 1584540694] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540781 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584540700.153330:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584540700.153331:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584540700.153332:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584540700.153346:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540700.153347:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540700.153348:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540700.153349:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540700.153350:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540700.153351:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584540700.153352:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584540700.153355:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22500 00000100:00080000:12.0:1584540700.153356:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584540700.153362:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540700.153363:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540700.153363:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540700.153365:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540700.153365:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540700.153366:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540700.153367:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540700.153368:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540700.157377:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584540700.157381:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584540700.157382:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584540700.157383:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584540700.157384:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584540700.157386:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584540700.157387:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584540700.157389:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540781 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584540700.157417:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078456880 new x1661492078456928 00000800:00000100:0.0:1584540706.141107:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540706.141662:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584540706.153713:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540700/real 1584540700] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540787 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584540706.153721:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584540706.153722:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584540706.153723:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584540706.153728:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540706.153730:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540706.153750:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540706.153751:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540706.153752:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540706.153754:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584540706.153755:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584540706.153756:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22506 00000100:00080000:12.0:1584540706.153758:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584540706.153763:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540706.153765:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540706.153765:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540706.153766:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540706.153767:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540706.153768:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540706.153769:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584540706.153770:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540706.157946:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584540706.157950:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584540706.157952:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584540706.157952:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584540706.157954:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584540706.157956:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584540706.157957:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584540706.157959:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540787 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540706.157967:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078456928 new x1661492078456976 00000800:00000100:0.0:1584540712.141094:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540712.141472:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540712.153522:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540706/real 1584540706] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540793 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540712.153530:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540712.153531:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540712.153532:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540712.153539:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540712.153541:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540712.153542:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540712.153543:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540712.153544:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540712.153545:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540712.153546:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540712.153547:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22512 00000100:00080000:14.0:1584540712.153549:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540712.153559:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540712.153560:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540712.153561:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540712.153562:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540712.153563:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540712.153564:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540712.153566:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540712.153567:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540712.157401:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584540712.157406:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584540712.157407:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584540712.157408:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584540712.157410:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584540712.157412:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584540712.157414:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584540712.157415:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540793 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540712.157444:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078456976 new x1661492078457024 00000800:00020000:2.0:1584540718.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584540718.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584540718.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540718.916493:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540718.928539:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540712/real 1584540712] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540799 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540718.928547:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540718.928548:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540718.928550:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540718.928555:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540718.928557:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540718.928558:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540718.928559:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540718.928559:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540718.928561:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540718.928561:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540718.928563:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22518 00000100:00080000:14.0:1584540718.928565:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540718.928573:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540718.928574:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540718.928575:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540718.928576:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540718.928578:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540718.928579:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540718.928580:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540718.928581:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540718.932450:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584540718.932454:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584540718.932456:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584540718.932457:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584540718.932458:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584540718.932460:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584540718.932461:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584540718.932463:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540799 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584540718.932493:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078457024 new x1661492078457056 00000800:00000100:0.0:1584540724.141258:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540724.141694:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584540724.153748:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540718/real 1584540718] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540805 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584540724.153759:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584540724.153760:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584540724.153762:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584540724.153781:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540724.153783:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540724.153784:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540724.153785:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540724.153786:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540724.153787:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584540724.153788:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584540724.153791:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22524 00000100:00080000:14.0:1584540724.153794:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584540724.153801:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540724.153803:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540724.153804:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540724.153806:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540724.153806:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540724.153808:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584540724.153808:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584540724.153809:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540724.157919:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584540724.157925:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584540724.157927:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584540724.157928:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584540724.157930:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584540724.157933:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584540724.157934:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584540724.157936:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540805 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584540724.157966:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078457056 new x1661492078457120 00000800:00000100:0.0:1584540730.140634:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540730.141056:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584540730.153132:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540724/real 1584540724] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540811 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584540730.153141:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584540730.153143:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584540730.153144:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584540730.153151:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540730.153154:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540730.153169:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540730.153170:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540730.153171:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540730.153173:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584540730.153174:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584540730.153176:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22530 00000100:00080000:16.0:1584540730.153178:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584540730.153196:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540730.153198:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540730.153198:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540730.153200:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540730.153200:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540730.153202:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540730.153202:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540730.153204:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540730.157344:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584540730.157350:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584540730.157352:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584540730.157353:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584540730.157356:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584540730.157358:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584540730.157359:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584540730.157361:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540811 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584540730.157391:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078457120 new x1661492078457168 00000800:00000100:0.0:1584540736.140497:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540736.141034:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584540736.153083:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540730/real 1584540730] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540817 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584540736.153091:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584540736.153093:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584540736.153095:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584540736.153102:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540736.153104:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540736.153105:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540736.153106:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540736.153107:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540736.153108:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584540736.153109:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584540736.153111:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22536 00000100:00080000:16.0:1584540736.153113:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584540736.153125:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540736.153126:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540736.153127:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540736.153128:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540736.153129:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540736.153130:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540736.153131:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540736.153132:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540736.157066:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584540736.157072:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584540736.157074:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584540736.157075:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584540736.157077:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584540736.157079:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584540736.157081:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584540736.157083:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540817 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584540736.157112:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078457168 new x1661492078457200 00000800:00020000:4.0:1584540742.916071:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584540742.916074:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584540742.916076:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540742.916509:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584540742.928560:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540736/real 1584540736] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540823 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584540742.928573:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584540742.928574:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584540742.928576:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584540742.928581:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540742.928583:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540742.928584:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540742.928585:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540742.928586:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540742.928587:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584540742.928588:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584540742.928590:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22542 00000100:00080000:16.0:1584540742.928592:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584540742.928602:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540742.928604:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540742.928605:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540742.928606:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540742.928606:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540742.928608:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540742.928608:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540742.928609:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540742.932668:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584540742.932672:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584540742.932674:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584540742.932675:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584540742.932676:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584540742.932678:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584540742.932680:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584540742.932682:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540823 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584540742.932711:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078457200 new x1661492078457248 00000800:00000100:0.0:1584540748.140673:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540748.141163:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584540748.153211:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540742/real 1584540742] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540829 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584540748.153219:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584540748.153220:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584540748.153221:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584540748.153234:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540748.153236:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540748.153237:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540748.153238:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540748.153238:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540748.153240:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584540748.153240:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584540748.153242:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22548 00000100:00080000:16.0:1584540748.153244:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584540748.153249:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540748.153251:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540748.153251:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540748.153252:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540748.153253:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540748.153254:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540748.153256:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540748.153257:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540748.157329:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584540748.157332:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584540748.157335:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584540748.157335:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584540748.157337:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584540748.157338:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584540748.157340:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584540748.157342:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540829 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584540748.157372:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078457248 new x1661492078457296 00000800:00000100:0.0:1584540754.140790:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540754.141215:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584540754.153267:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540748/real 1584540748] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540835 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584540754.153275:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584540754.153276:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584540754.153278:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584540754.153283:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540754.153285:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540754.153294:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540754.153296:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540754.153296:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540754.153298:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584540754.153298:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584540754.153300:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22554 00000100:00080000:16.0:1584540754.153302:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584540754.153308:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540754.153309:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540754.153309:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540754.153311:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540754.153312:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540754.153313:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540754.153314:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540754.153315:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540754.157237:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584540754.157240:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584540754.157242:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584540754.157243:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584540754.157244:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584540754.157246:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584540754.157247:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584540754.157249:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540835 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584540754.157278:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078457296 new x1661492078457360 00000800:00000100:0.0:1584540760.140785:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540760.141192:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584540760.153240:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540754/real 1584540754] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540841 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584540760.153247:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584540760.153248:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584540760.153250:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584540760.153255:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540760.153257:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540760.153257:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540760.153259:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540760.153259:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540760.153261:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584540760.153261:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584540760.153263:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22560 00000100:00080000:16.0:1584540760.153265:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584540760.153274:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540760.153275:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540760.153277:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540760.153278:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540760.153278:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540760.153279:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540760.153280:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540760.153281:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540760.157305:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584540760.157309:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584540760.157310:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584540760.157311:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584540760.157312:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584540760.157314:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584540760.157316:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584540760.157317:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540841 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584540760.157347:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078457360 new x1661492078457392 00000800:00020000:2.0:1584540766.916073:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584540766.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584540766.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540766.916651:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584540766.928715:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540760/real 1584540760] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540847 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584540766.928723:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584540766.928724:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584540766.928726:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584540766.928731:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540766.928733:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540766.928734:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540766.928735:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540766.928735:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540766.928737:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584540766.928737:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584540766.928739:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22566 00000100:00080000:16.0:1584540766.928741:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584540766.928751:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540766.928752:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540766.928753:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540766.928754:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540766.928755:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540766.928756:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584540766.928756:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584540766.928757:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540766.932635:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540766.932640:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540766.932641:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540766.932642:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540766.932643:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540766.932646:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540766.932648:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540766.932650:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540847 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584540766.932660:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078457392 new x1661492078457440 00000800:00000100:0.0:1584540772.141005:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540772.141497:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584540772.153548:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540766/real 1584540766] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540853 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584540772.153556:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584540772.153558:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584540772.153559:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584540772.153576:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540772.153578:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540772.153579:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540772.153580:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540772.153580:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540772.153582:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584540772.153583:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584540772.153585:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22572 00000100:00080000:18.0:1584540772.153588:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584540772.153594:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540772.153595:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540772.153596:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540772.153597:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540772.153598:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540772.153599:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540772.153600:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540772.153601:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540772.157582:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540772.157587:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540772.157588:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540772.157589:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540772.157590:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540772.157592:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540772.157594:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540772.157596:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540853 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584540772.157604:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078457440 new x1661492078457488 00000800:00000100:0.0:1584540778.141047:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540778.141487:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584540778.153544:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540772/real 1584540772] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540859 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584540778.153552:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584540778.153553:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584540778.153554:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584540778.153560:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540778.153562:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540778.153571:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540778.153572:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540778.153573:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540778.153574:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584540778.153576:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584540778.153577:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22578 00000100:00080000:18.0:1584540778.153579:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584540778.153585:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540778.153586:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540778.153587:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540778.153588:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540778.153589:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540778.153590:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540778.153590:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540778.153592:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540778.157319:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540778.157324:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540778.157325:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540778.157326:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540778.157327:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540778.157330:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540778.157331:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540778.157333:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540859 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584540778.157342:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078457488 new x1661492078457536 00000800:00000100:0.0:1584540784.141220:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540784.141661:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584540784.153716:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540778/real 1584540778] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540865 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584540784.153727:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584540784.153728:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584540784.153730:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584540784.153737:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540784.153740:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540784.153741:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540784.153742:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540784.153744:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540784.153745:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584540784.153746:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584540784.153749:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22584 00000100:00080000:18.0:1584540784.153752:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584540784.153768:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540784.153770:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540784.153770:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540784.153772:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540784.153773:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540784.153774:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540784.153775:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540784.153776:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540784.157707:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540784.157714:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540784.157717:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540784.157718:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540784.157720:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540784.157723:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540784.157725:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540784.157727:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540865 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584540784.157736:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078457536 new x1661492078457584 00000800:00020000:2.0:1584540790.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584540790.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584540790.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540790.916539:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584540790.928600:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540784/real 1584540784] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540871 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584540790.928612:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584540790.928614:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584540790.928616:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584540790.928624:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540790.928626:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540790.928628:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540790.928630:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540790.928630:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540790.928632:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584540790.928633:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584540790.928635:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22590 00000100:00080000:18.0:1584540790.928639:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584540790.928656:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540790.928657:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540790.928658:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540790.928659:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540790.928660:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540790.928661:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540790.928662:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540790.928663:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540790.932745:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540790.932753:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540790.932756:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540790.932757:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540790.932759:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540790.932762:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540790.932763:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540790.932765:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540871 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584540790.932795:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078457584 new x1661492078457616 00000800:00000100:0.0:1584540796.141534:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540796.142057:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584540796.154108:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540790/real 1584540790] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540877 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584540796.154116:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584540796.154117:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584540796.154118:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584540796.154134:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540796.154136:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540796.154137:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540796.154138:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540796.154139:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540796.154140:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584540796.154141:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584540796.154142:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22596 00000100:00080000:18.0:1584540796.154144:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584540796.154150:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540796.154151:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540796.154152:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540796.154153:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540796.154153:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540796.154154:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540796.154155:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540796.154156:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540796.158011:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540796.158015:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540796.158016:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540796.158017:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540796.158018:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540796.158020:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540796.158022:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540796.158024:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540877 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584540796.158074:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078457616 new x1661492078457664 00000800:00000100:0.0:1584540802.140630:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540802.141042:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584540802.153093:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540796/real 1584540796] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540883 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584540802.153101:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584540802.153102:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584540802.153104:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584540802.153110:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540802.153111:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540802.153121:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540802.153123:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540802.153123:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540802.153125:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584540802.153125:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584540802.153127:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22602 00000100:00080000:18.0:1584540802.153129:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584540802.153135:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540802.153136:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540802.153136:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540802.153138:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540802.153138:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540802.153139:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540802.153140:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540802.153141:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540802.156910:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540802.156914:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540802.156916:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540802.156917:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540802.156918:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540802.156921:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540802.156922:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540802.156924:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540883 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584540802.156953:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078457664 new x1661492078457728 00000800:00000100:0.0:1584540808.140788:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540808.141201:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584540808.153249:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540802/real 1584540802] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540889 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584540808.153258:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584540808.153259:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584540808.153260:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584540808.153266:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540808.153268:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540808.153269:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540808.153270:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540808.153271:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540808.153272:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584540808.153273:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584540808.153274:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22608 00000100:00080000:18.0:1584540808.153276:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584540808.153284:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540808.153286:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540808.153286:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540808.153287:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540808.153288:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540808.153289:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540808.153290:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540808.153291:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540808.157258:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540808.157261:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540808.157263:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540808.157264:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540808.157265:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540808.157267:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540808.157268:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540808.157270:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540889 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584540808.157299:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078457728 new x1661492078457760 00000800:00020000:2.0:1584540814.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584540814.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584540814.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540814.916543:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584540814.928632:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540808/real 1584540808] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540895 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584540814.972524:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584540814.972525:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584540814.972538:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584540814.972542:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540814.972544:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540814.972544:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540814.972546:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540814.972546:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540814.972547:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584540814.972548:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584540814.972549:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22614 00000100:00080000:18.0:1584540814.972551:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584540814.972559:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540814.972560:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540814.972561:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540814.972562:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540814.972563:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540814.972564:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540814.972565:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540814.972577:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540814.976450:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540814.976454:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540814.976456:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540814.976457:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540814.976458:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540814.976460:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540814.976461:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540814.976463:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540895 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584540814.976492:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078457760 new x1661492078457808 00000100:00080000:2.0:1584540819.964943:0:178782:0:(client.c:881:ptlrpc_request_alloc_internal()) public1-OST0000-osc-ffff8d3ae8ab1800: reconnect after 8037s idle 00000100:00080000:2.0:1584540819.964945:0:178782:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf2800 public1-OST0000_UUID: changing import state from NEW to CONNECTING 00000100:00080000:2.0:1584540819.964947:0:178782:0:(import.c:528:import_select_connection()) public1-OST0000-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 14555 00000100:00080000:2.0:1584540819.964949:0:178782:0:(import.c:604:import_select_connection()) public1-OST0000-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf2800 using connection 10.10.2.21@o2ib/10.10.2.21@o2ib 00000100:00080000:2.0:1584540819.964960:0:178782:0:(client.c:1575:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8d3396aca880 x1661492078457856/t0(0) o13->public1-OST0000-osc-ffff8d3ae8ab1800@10.10.2.21@o2ib:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 00000100:00080000:20.0:1584540819.965214:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0000-osc-ffff8d3ae8ab1800: connect to target with instance 4 00000100:00080000:20.0:1584540819.965217:0:25724:0:(import.c:907:ptlrpc_connect_set_flags()) public1-OST0000-osc-ffff8d3ae8ab1800: Resetting ns_connect_flags to server flags: 0xa0425af2e3440478 00000080:00080000:20.0:1584540819.965220:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540819.965222:0:25724:0:(import.c:1132:ptlrpc_connect_interpret()) connected to replayable target: public1-OST0000_UUID 00000100:00080000:20.0:1584540819.965222:0:25724:0:(import.c:1152:ptlrpc_connect_interpret()) ffff8d3ac9bf2800 public1-OST0000_UUID: changing import state from CONNECTING to FULL 00000080:00080000:20.0:1584540819.965225:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540819.965227:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32f2ba3a80): req@ffff8d3396aca880 x1661492078457856/t0(0) o13->public1-OST0000-osc-ffff8d3ae8ab1800@10.10.2.21@o2ib:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 00000800:00000100:0.0:1584540820.140958:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540820.141360:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584540820.153409:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540814/real 1584540814] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540901 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584540820.153417:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584540820.153418:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584540820.153420:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584540820.153433:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540820.153434:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540820.153435:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540820.153437:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584540820.153437:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584540820.153439:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22620 00000100:00080000:18.0:1584540820.153441:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584540820.153446:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540820.153447:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540820.153448:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540820.153449:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540820.153450:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540820.153451:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540820.153452:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540820.153453:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540820.157268:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540820.157272:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540820.157274:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540820.157275:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540820.157276:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540820.157278:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540820.157279:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540820.157281:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540901 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584540820.157290:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078457808 new x1661492078457920 00000800:00000100:0.0:1584540826.140900:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540826.141445:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584540826.153486:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540820/real 1584540820] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540907 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584540826.153494:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584540826.153495:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584540826.153496:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584540826.153502:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540826.153503:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540826.153504:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540826.153505:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584540826.153506:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584540826.153507:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22626 00000100:00080000:18.0:1584540826.153509:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584540826.153518:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540826.153519:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540826.153519:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540826.153521:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540826.153521:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540826.153522:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540826.153524:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540826.153525:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540826.157377:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540826.157381:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540826.157382:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540826.157383:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540826.157384:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540826.157386:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540826.157388:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540826.157390:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540907 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584540826.157398:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078457920 new x1661492078457952 00000100:00000400:2.0:1584540831.965071:0:178782:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584540819/real 1584540819] req@ffff8d3396aca880 x1661492078457872/t0(0) o13->public1-OST0001-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:7/4 lens 224/368 e 0 to 1 dl 1584540831 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 00000100:02000400:2.0:1584540831.965078:0:178782:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection to public1-OST0001 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584540831.965080:0:178782:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584540831.965081:0:178782:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0001_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584540831.965087:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540831.965089:0:176857:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:2.0:1584540831.965090:0:176857:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584540831.965091:0:176857:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22344 00000100:00080000:2.0:1584540831.965094:0:176857:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584540831.965103:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540831.965105:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540831.965105:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540831.965108:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540831.965108:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540831.965109:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540831.965110:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540831.965111:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584540831.965112:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584540831.965113:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000800:00000100:0.0:1584540832.140984:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540832.141475:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584540832.153529:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540826/real 1584540826] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540913 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584540832.153536:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584540832.153538:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584540832.153539:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584540832.153545:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540832.153547:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540832.153548:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540832.153549:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584540832.153550:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584540832.153551:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22632 00000100:00080000:18.0:1584540832.153553:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584540832.153562:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540832.153563:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540832.153564:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540832.153565:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540832.153566:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540832.153567:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540832.153569:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540832.153570:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540832.157533:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540832.157537:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540832.157539:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540832.157540:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540832.157541:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540832.157543:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540832.157546:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540832.157548:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540913 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584540832.157555:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078457952 new x1661492078458016 00000100:00080000:20.0:1584540837.153170:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:20.0:1584540837.153171:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0001_UUID on 10.10.2.22@o2ib failed (-110) 00000800:00020000:4.0:1584540838.916123:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584540838.916128:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584540838.916133:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540838.916570:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584540838.928654:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540832/real 1584540832] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540919 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584540838.928672:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584540838.954148:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584540838.954149:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584540838.954153:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540838.954155:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540838.954155:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540838.954157:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584540838.954157:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584540838.954159:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22638 00000100:00080000:18.0:1584540838.954160:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584540838.954168:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540838.954169:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540838.954170:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540838.954171:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540838.954171:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540838.954172:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584540838.954185:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584540838.954186:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540838.958170:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540838.958174:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540838.958175:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540838.958176:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540838.958177:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540838.958179:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540838.958180:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540838.976067:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540919 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584540838.976111:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078458016 new x1661492078458048 00000800:00000100:0.0:1584540844.141161:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540844.141599:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584540844.153646:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540838/real 1584540838] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540925 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584540844.153659:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584540844.153660:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584540844.153662:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584540844.153679:0:4063:0:(import.c:1607:ptlrpc_disconnect_prep_req()) ffff8d3ac9bf2800 public1-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:22.0:1584540844.153680:0:4063:0:(import.c:1764:ptlrpc_disconnect_and_idle_import()) public1-OST0000-osc-ffff8d3ae8ab1800: disconnect after 25s idle 00000100:00080000:22.0:1584540844.153683:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540844.153685:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540844.153686:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540844.153687:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584540844.153688:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584540844.153689:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22644 00000100:00080000:22.0:1584540844.153692:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584540844.153696:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540844.153698:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540844.153698:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540844.153700:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540844.153700:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540844.153701:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540844.153702:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540844.153703:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540844.153864:0:25724:0:(import.c:1708:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=5: rc = 0 req@ffff8d3396b6c800 x1661492078458096/t0(0) o9->public1-OST0000-osc-ffff8d3ae8ab1800@10.10.2.21@o2ib:28/4 lens 224/192 e 0 to 0 dl 1584540850 ref 1 fl Interpret:RN/0/0 rc 0/0 00000100:00080000:20.0:1584540844.157697:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540844.157702:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540844.157703:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540844.157704:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540844.157705:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540844.157707:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540844.157709:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540844.157710:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540925 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584540844.157739:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078458048 new x1661492078458128 00000800:00000100:0.0:1584540850.141356:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540850.141783:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584540850.153835:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540844/real 1584540844] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540931 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584540850.153844:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584540850.153846:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584540850.153847:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584540850.153853:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540850.153855:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540850.153856:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540850.153857:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584540850.153858:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584540850.153860:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22650 00000100:00080000:22.0:1584540850.153863:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584540850.153877:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540850.153879:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540850.153879:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540850.153881:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540850.153881:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540850.153883:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540850.153884:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540850.153885:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540850.157857:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540850.157863:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540850.157866:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540850.157867:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540850.157869:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540850.157871:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540850.157873:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540850.157875:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540931 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584540850.157883:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078458128 new x1661492078458160 00000800:00000100:0.0:1584540856.139777:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540856.140361:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584540856.152411:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540850/real 1584540850] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540937 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584540856.152420:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584540856.152422:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584540856.152423:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584540856.152429:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:22.0:1584540856.152432:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:22.0:1584540856.152433:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584540856.152435:0:4063:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22637 00000100:00080000:22.0:1584540856.152436:0:4063:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 21221 00000100:00080000:22.0:1584540856.152439:0:4063:0:(import.c:596:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0001 (at 10.10.2.21@o2ib) 00000100:00080000:22.0:1584540856.152440:0:4063:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.21@o2ib/10.10.2.21@o2ib 00000100:00080000:22.0:1584540856.152455:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540856.152456:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540856.152457:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540856.152458:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584540856.152459:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584540856.152460:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22656 00000100:00080000:22.0:1584540856.152462:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584540856.152465:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540856.152467:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540856.152468:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540856.152470:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540856.152470:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540856.152472:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540856.152472:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540856.152474:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540856.152671:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:20.0:1584540856.152673:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0001_UUID on 10.10.2.21@o2ib failed (-19) 00000100:00080000:20.0:1584540856.156279:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540856.156285:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540856.156287:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540856.156288:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540856.156290:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540856.156292:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540856.156294:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540856.156296:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540937 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584540856.156304:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078458160 new x1661492078458208 00000800:00020000:4.0:1584540862.916055:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584540862.916064:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584540862.916068:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540862.916472:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584540862.928560:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540856/real 1584540856] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540943 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584540862.928582:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584540862.928584:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584540862.928586:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584540862.928594:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540862.928596:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540862.928597:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540862.928599:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584540862.928600:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584540862.928602:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22662 00000100:00080000:22.0:1584540862.928606:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584540862.928623:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540862.928624:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540862.928626:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540862.928627:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540862.928628:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540862.928629:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540862.928641:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540862.928642:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540862.932656:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540862.932664:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540862.932667:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540862.932668:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540862.932670:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540862.932673:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540862.932674:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540862.932676:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540943 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584540862.932727:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078458208 new x1661492078458256 00000800:00000100:0.0:1584540868.133901:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540868.134314:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584540868.146359:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540862/real 1584540862] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540949 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584540868.146368:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584540868.146369:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584540868.146370:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584540868.146387:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540868.146389:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540868.146390:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540868.146391:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540868.146392:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540868.146393:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584540868.146394:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584540868.146396:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22668 00000100:00080000:22.0:1584540868.146399:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584540868.146407:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540868.146408:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540868.146409:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540868.146410:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540868.146411:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540868.146412:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540868.146413:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540868.146414:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540868.150270:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540868.150276:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540868.150278:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540868.150279:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540868.150280:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540868.150283:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540868.150284:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540868.150286:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540949 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584540868.150295:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078458256 new x1661492078458304 00000800:00000100:0.0:1584540874.131602:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540874.132065:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584540874.144152:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540868/real 1584540868] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540955 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584540874.144161:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584540874.144162:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584540874.144163:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584540874.144169:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540874.144171:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540874.144172:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540874.144173:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540874.144174:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540874.144175:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584540874.144176:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584540874.144178:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22674 00000100:00080000:22.0:1584540874.144181:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584540874.144191:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540874.144192:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540874.144193:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540874.144194:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540874.144195:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540874.144196:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540874.144197:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540874.144198:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540874.148244:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540874.148249:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540874.148251:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540874.148252:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540874.148253:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540874.148256:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540874.148257:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540874.148259:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540955 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584540874.148288:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078458304 new x1661492078458352 00000800:00000100:0.0:1584540880.131680:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540880.132079:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584540880.144128:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540874/real 1584540874] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540961 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584540880.144136:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584540880.144137:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584540880.144139:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584540880.144144:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540880.144145:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540880.144146:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:22.0:1584540880.144147:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:22.0:1584540880.144148:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584540880.144150:0:4063:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22637 00000100:00080000:22.0:1584540880.144152:0:4063:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 22662 00000100:00080000:22.0:1584540880.144153:0:4063:0:(import.c:570:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: tried all connections, increasing latency to 6s 00000100:00080000:22.0:1584540880.144155:0:4063:0:(import.c:596:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0001 (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584540880.144156:0:4063:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584540880.144165:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540880.144166:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540880.144167:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540880.144168:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584540880.144168:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584540880.144170:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22680 00000100:00080000:22.0:1584540880.144171:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584540880.144174:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540880.144175:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540880.144176:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540880.144177:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540880.144178:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540880.144179:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540880.144180:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540880.144181:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584540880.148159:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540880.148163:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540880.148165:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0001_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540880.148165:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540880.148167:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540880.148169:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540880.148170:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540880.148172:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32f2ba3a80): req@ffff8d3396aca880 x1661492078457872/t0(0) o13->public1-OST0001-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:7/4 lens 224/368 e 0 to 1 dl 1584540831 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 00000100:00080000:20.0:1584540880.148181:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584540880.148183:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540880.148184:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584540880.148184:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584540880.148185:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584540880.148186:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584540880.148187:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584540880.148189:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540961 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584540880.148217:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078458352 new x1661492078458400 00000100:00080000:2.0:1584540880.148320:0:178782:0:(client.c:881:ptlrpc_request_alloc_internal()) public1-OST0002-osc-ffff8d3ae8ab1800: reconnect after 8098s idle 00000100:00080000:2.0:1584540880.148322:0:178782:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f800 public1-OST0002_UUID: changing import state from NEW to CONNECTING 00000100:00080000:2.0:1584540880.148324:0:178782:0:(import.c:528:import_select_connection()) public1-OST0002-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 14562 00000100:00080000:2.0:1584540880.148327:0:178782:0:(import.c:604:import_select_connection()) public1-OST0002-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f800 using connection 10.10.2.21@o2ib/10.10.2.21@o2ib 00000100:00080000:2.0:1584540880.148342:0:178782:0:(client.c:1575:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8d3396aca880 x1661492078458432/t0(0) o13->public1-OST0002-osc-ffff8d3ae8ab1800@10.10.2.21@o2ib:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 00000100:00080000:20.0:1584540880.148596:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0002-osc-ffff8d3ae8ab1800: connect to target with instance 3 00000100:00080000:20.0:1584540880.148598:0:25724:0:(import.c:907:ptlrpc_connect_set_flags()) public1-OST0002-osc-ffff8d3ae8ab1800: Resetting ns_connect_flags to server flags: 0xa0425af2e3440478 00000080:00080000:20.0:1584540880.148600:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540880.148601:0:25724:0:(import.c:1132:ptlrpc_connect_interpret()) connected to replayable target: public1-OST0002_UUID 00000100:00080000:20.0:1584540880.148601:0:25724:0:(import.c:1152:ptlrpc_connect_interpret()) ffff8d31d6e0f800 public1-OST0002_UUID: changing import state from CONNECTING to FULL 00000080:00080000:20.0:1584540880.148603:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540880.148604:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ec67e380): req@ffff8d3396aca880 x1661492078458432/t0(0) o13->public1-OST0002-osc-ffff8d3ae8ab1800@10.10.2.21@o2ib:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 00000100:00080000:4.0:1584540880.148853:0:178782:0:(client.c:881:ptlrpc_request_alloc_internal()) public1-OST0004-osc-ffff8d3ae8ab1800: reconnect after 8098s idle 00000100:00080000:4.0:1584540880.148855:0:178782:0:(import.c:684:ptlrpc_connect_import()) ffff8d3acb66c800 public1-OST0004_UUID: changing import state from NEW to CONNECTING 00000100:00080000:4.0:1584540880.148857:0:178782:0:(import.c:528:import_select_connection()) public1-OST0004-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 14566 00000100:00080000:4.0:1584540880.148859:0:178782:0:(import.c:604:import_select_connection()) public1-OST0004-osc-ffff8d3ae8ab1800: import ffff8d3acb66c800 using connection 10.10.2.21@o2ib/10.10.2.21@o2ib 00000100:00080000:4.0:1584540880.148869:0:178782:0:(client.c:1575:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8d2babe45a00 x1661492078458480/t0(0) o13->public1-OST0004-osc-ffff8d3ae8ab1800@10.10.2.21@o2ib:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 00000100:00080000:20.0:1584540880.149060:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0004-osc-ffff8d3ae8ab1800: connect to target with instance 3 00000100:00080000:20.0:1584540880.149062:0:25724:0:(import.c:907:ptlrpc_connect_set_flags()) public1-OST0004-osc-ffff8d3ae8ab1800: Resetting ns_connect_flags to server flags: 0xa0425af2e3440478 00000080:00080000:20.0:1584540880.149065:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540880.149066:0:25724:0:(import.c:1132:ptlrpc_connect_interpret()) connected to replayable target: public1-OST0004_UUID 00000100:00080000:20.0:1584540880.149067:0:25724:0:(import.c:1152:ptlrpc_connect_interpret()) ffff8d3acb66c800 public1-OST0004_UUID: changing import state from CONNECTING to FULL 00000080:00080000:20.0:1584540880.149069:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540880.149071:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d31d42fc580): req@ffff8d2babe45a00 x1661492078458480/t0(0) o13->public1-OST0004-osc-ffff8d3ae8ab1800@10.10.2.21@o2ib:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 00000100:00080000:4.0:1584540880.149198:0:178782:0:(client.c:881:ptlrpc_request_alloc_internal()) public1-OST0005-osc-ffff8d3ae8ab1800: reconnect after 8098s idle 00000100:00080000:4.0:1584540880.149200:0:178782:0:(import.c:684:ptlrpc_connect_import()) ffff8d3acb66b000 public1-OST0005_UUID: changing import state from NEW to CONNECTING 00000100:00080000:4.0:1584540880.149202:0:178782:0:(import.c:528:import_select_connection()) public1-OST0005-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14562 00000100:00080000:4.0:1584540880.149203:0:178782:0:(import.c:604:import_select_connection()) public1-OST0005-osc-ffff8d3ae8ab1800: import ffff8d3acb66b000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584540880.149211:0:178782:0:(client.c:1575:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8d2babe45a00 x1661492078458512/t0(0) o13->public1-OST0005-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 00000100:00080000:20.0:1584540880.149365:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0005-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000100:00080000:20.0:1584540880.149368:0:25724:0:(import.c:907:ptlrpc_connect_set_flags()) public1-OST0005-osc-ffff8d3ae8ab1800: Resetting ns_connect_flags to server flags: 0xa0425af2e3440478 00000080:00080000:20.0:1584540880.149369:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540880.149370:0:25724:0:(import.c:1132:ptlrpc_connect_interpret()) connected to replayable target: public1-OST0005_UUID 00000100:00080000:20.0:1584540880.149371:0:25724:0:(import.c:1152:ptlrpc_connect_interpret()) ffff8d3acb66b000 public1-OST0005_UUID: changing import state from CONNECTING to FULL 00000080:00080000:20.0:1584540880.149372:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540880.149373:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d31d42fc580): req@ffff8d2babe45a00 x1661492078458512/t0(0) o13->public1-OST0005-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 00000100:00080000:4.0:1584540880.149489:0:178782:0:(client.c:881:ptlrpc_request_alloc_internal()) public1-OST0006-osc-ffff8d3ae8ab1800: reconnect after 8098s idle 00000100:00080000:4.0:1584540880.149490:0:178782:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0e000 public1-OST0006_UUID: changing import state from NEW to CONNECTING 00000100:00080000:4.0:1584540880.149492:0:178782:0:(import.c:528:import_select_connection()) public1-OST0006-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 14557 00000100:00080000:4.0:1584540880.149493:0:178782:0:(import.c:604:import_select_connection()) public1-OST0006-osc-ffff8d3ae8ab1800: import ffff8d31d6e0e000 using connection 10.10.2.21@o2ib/10.10.2.21@o2ib 00000100:00080000:4.0:1584540880.149501:0:178782:0:(client.c:1575:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8d2babe45a00 x1661492078458544/t0(0) o13->public1-OST0006-osc-ffff8d3ae8ab1800@10.10.2.21@o2ib:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 00000100:00080000:20.0:1584540880.149645:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0006-osc-ffff8d3ae8ab1800: connect to target with instance 3 00000100:00080000:20.0:1584540880.149646:0:25724:0:(import.c:907:ptlrpc_connect_set_flags()) public1-OST0006-osc-ffff8d3ae8ab1800: Resetting ns_connect_flags to server flags: 0xa0425af2e3440478 00000080:00080000:20.0:1584540880.149648:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540880.149648:0:25724:0:(import.c:1132:ptlrpc_connect_interpret()) connected to replayable target: public1-OST0006_UUID 00000100:00080000:20.0:1584540880.149649:0:25724:0:(import.c:1152:ptlrpc_connect_interpret()) ffff8d31d6e0e000 public1-OST0006_UUID: changing import state from CONNECTING to FULL 00000080:00080000:20.0:1584540880.149650:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540880.149652:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d31d42fc580): req@ffff8d2babe45a00 x1661492078458544/t0(0) o13->public1-OST0006-osc-ffff8d3ae8ab1800@10.10.2.21@o2ib:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 00000100:00080000:4.0:1584540880.149775:0:178782:0:(client.c:881:ptlrpc_request_alloc_internal()) public1-OST0007-osc-ffff8d3ae8ab1800: reconnect after 8098s idle 00000100:00080000:4.0:1584540880.149776:0:178782:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0e800 public1-OST0007_UUID: changing import state from NEW to CONNECTING 00000100:00080000:4.0:1584540880.149777:0:178782:0:(import.c:528:import_select_connection()) public1-OST0007-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 14557 00000100:00080000:4.0:1584540880.149779:0:178782:0:(import.c:604:import_select_connection()) public1-OST0007-osc-ffff8d3ae8ab1800: import ffff8d31d6e0e800 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584540880.149786:0:178782:0:(client.c:1575:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8d2babe45a00 x1661492078458576/t0(0) o13->public1-OST0007-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 00000100:00080000:20.0:1584540880.149903:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0007-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000100:00080000:20.0:1584540880.149905:0:25724:0:(import.c:907:ptlrpc_connect_set_flags()) public1-OST0007-osc-ffff8d3ae8ab1800: Resetting ns_connect_flags to server flags: 0xa0425af2e3440478 00000080:00080000:20.0:1584540880.149906:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540880.149907:0:25724:0:(import.c:1132:ptlrpc_connect_interpret()) connected to replayable target: public1-OST0007_UUID 00000100:00080000:20.0:1584540880.149907:0:25724:0:(import.c:1152:ptlrpc_connect_interpret()) ffff8d31d6e0e800 public1-OST0007_UUID: changing import state from CONNECTING to FULL 00000080:00080000:20.0:1584540880.149909:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584540880.149910:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d31d42fc580): req@ffff8d2babe45a00 x1661492078458576/t0(0) o13->public1-OST0007-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 00000800:00020000:4.0:1584540886.916133:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584540886.916139:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584540886.916143:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540886.916605:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584540886.928688:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540880/real 1584540880] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540967 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584540886.928706:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584540886.928707:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584540886.928708:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584540886.928714:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540886.928716:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540886.928716:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540886.928718:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584540886.928718:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584540886.928720:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22686 00000100:00080000:22.0:1584540886.928722:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584540886.932492:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584540886.932497:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584540886.932498:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584540886.932499:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584540886.932500:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584540886.932502:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584540886.932504:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584540886.932506:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540967 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584540886.932515:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078458400 new x1661492078458624 00000800:00000100:0.0:1584540892.133160:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540892.133587:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584540892.145639:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540886/real 1584540886] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540973 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584540892.145647:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584540892.145648:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584540892.145649:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584540892.145663:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540892.145665:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584540892.145665:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584540892.145667:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584540892.145668:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584540892.145670:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22692 00000100:00080000:22.0:1584540892.145672:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584540892.149514:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584540892.149519:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584540892.149520:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584540892.149521:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584540892.149522:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584540892.149524:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584540892.149525:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584540892.149527:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540973 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584540892.149556:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078458624 new x1661492078458672 00000800:00000100:0.0:1584540898.134486:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540898.134917:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584540898.146991:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540892/real 1584540892] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540979 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584540898.146999:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584540898.147000:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584540898.147001:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584540898.147007:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540898.147009:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584540898.147009:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584540898.147011:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584540898.147011:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584540898.147013:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22698 00000100:00080000:6.0:1584540898.147015:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584540898.151099:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584540898.151103:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584540898.151104:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584540898.151105:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584540898.151107:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584540898.151108:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584540898.151110:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584540898.151112:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540979 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584540898.151123:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078458672 new x1661492078458704 00000800:00000100:0.0:1584540904.135679:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540904.136174:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584540904.148270:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540898/real 1584540898] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540985 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584540904.148281:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584540904.148282:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584540904.148285:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584540904.148293:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540904.148295:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540904.148311:0:171:0:(import.c:1607:ptlrpc_disconnect_prep_req()) ffff8d31d6e0f800 public1-OST0002_UUID: changing import state from FULL to CONNECTING 00000100:00080000:8.0:1584540904.148313:0:171:0:(import.c:1764:ptlrpc_disconnect_and_idle_import()) public1-OST0002-osc-ffff8d3ae8ab1800: disconnect after 24s idle 00000100:00080000:8.0:1584540904.148316:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584540904.148319:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584540904.148332:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584540904.148334:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22704 00000100:00080000:8.0:1584540904.148338:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584540904.148346:0:171:0:(import.c:1607:ptlrpc_disconnect_prep_req()) ffff8d3acb66c800 public1-OST0004_UUID: changing import state from FULL to CONNECTING 00000100:00080000:8.0:1584540904.148347:0:171:0:(import.c:1764:ptlrpc_disconnect_and_idle_import()) public1-OST0004-osc-ffff8d3ae8ab1800: disconnect after 24s idle 00000100:00080000:8.0:1584540904.148350:0:171:0:(import.c:1607:ptlrpc_disconnect_prep_req()) ffff8d3acb66b000 public1-OST0005_UUID: changing import state from FULL to CONNECTING 00000100:00080000:8.0:1584540904.148351:0:171:0:(import.c:1764:ptlrpc_disconnect_and_idle_import()) public1-OST0005-osc-ffff8d3ae8ab1800: disconnect after 24s idle 00000100:00080000:8.0:1584540904.148353:0:171:0:(import.c:1607:ptlrpc_disconnect_prep_req()) ffff8d31d6e0e000 public1-OST0006_UUID: changing import state from FULL to CONNECTING 00000100:00080000:8.0:1584540904.148354:0:171:0:(import.c:1764:ptlrpc_disconnect_and_idle_import()) public1-OST0006-osc-ffff8d3ae8ab1800: disconnect after 24s idle 00000100:00080000:8.0:1584540904.148356:0:171:0:(import.c:1607:ptlrpc_disconnect_prep_req()) ffff8d31d6e0e800 public1-OST0007_UUID: changing import state from FULL to CONNECTING 00000100:00080000:8.0:1584540904.148357:0:171:0:(import.c:1764:ptlrpc_disconnect_and_idle_import()) public1-OST0007-osc-ffff8d3ae8ab1800: disconnect after 24s idle 00000100:00080000:6.0:1584540904.148536:0:25724:0:(import.c:1708:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=5: rc = 0 req@ffff8d32ff3a5e80 x1661492078458816/t0(0) o9->public1-OST0006-osc-ffff8d3ae8ab1800@10.10.2.21@o2ib:28/4 lens 224/192 e 0 to 0 dl 1584540910 ref 1 fl Interpret:RN/0/0 rc 0/0 00000100:00080000:6.0:1584540904.148543:0:25724:0:(import.c:1708:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=5: rc = 0 req@ffff8d32ff3a7080 x1661492078458784/t0(0) o9->public1-OST0004-osc-ffff8d3ae8ab1800@10.10.2.21@o2ib:28/4 lens 224/192 e 0 to 0 dl 1584540910 ref 1 fl Interpret:RN/0/0 rc 0/0 00000100:00080000:6.0:1584540904.148549:0:25724:0:(import.c:1708:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=5: rc = 0 req@ffff8d32ff3a4c80 x1661492078458752/t0(0) o9->public1-OST0002-osc-ffff8d3ae8ab1800@10.10.2.21@o2ib:28/4 lens 224/192 e 0 to 0 dl 1584540910 ref 1 fl Interpret:RN/0/0 rc 0/0 00000100:00080000:6.0:1584540904.152333:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:6.0:1584540904.152341:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:6.0:1584540904.152343:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:6.0:1584540904.152344:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:6.0:1584540904.152346:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:6.0:1584540904.152349:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:6.0:1584540904.152350:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584540904.152352:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540985 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584540904.152363:0:25724:0:(import.c:1708:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=5: rc = 0 req@ffff8d32ff3a2880 x1661492078458832/t0(0) o9->public1-OST0007-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 224/192 e 0 to 0 dl 1584540910 ref 1 fl Interpret:RN/0/0 rc 0/0 00000100:00080000:6.0:1584540904.152369:0:25724:0:(import.c:1708:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=5: rc = 0 req@ffff8d32ff3a5a00 x1661492078458800/t0(0) o9->public1-OST0005-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 224/192 e 0 to 0 dl 1584540910 ref 1 fl Interpret:RN/0/0 rc 0/0 00000100:00080000:10.0:1584540904.152384:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078458704 new x1661492078458848 00000800:00020000:4.0:1584540910.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584540910.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584540910.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540910.916580:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540910.928676:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540904/real 1584540904] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540991 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540910.928697:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540910.928698:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540910.928701:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540910.928708:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540910.928710:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540910.928711:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540910.928713:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540910.928715:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540910.928717:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22710 00000100:00080000:10.0:1584540910.928720:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584540910.933023:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584540910.933035:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584540910.933048:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584540910.933049:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584540910.933051:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584540910.933054:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584540910.933055:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584540910.933057:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540991 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584540910.933086:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078458848 new x1661492078458880 00000800:00000100:0.0:1584540916.139107:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540916.139602:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540916.151684:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540910/real 1584540910] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540997 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540916.151692:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540916.151693:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540916.151695:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540916.151710:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540916.151712:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540916.151713:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540916.151714:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540916.151715:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540916.151716:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22716 00000100:00080000:10.0:1584540916.151718:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584540919.457979:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584540919.457988:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584540919.457990:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584540919.457992:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584540919.457994:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584540919.457997:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584540919.457999:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584540919.458001:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584540997 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584540919.458014:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078458880 new x1661492078458944 00000800:00020000:4.0:1584540924.916218:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584540924.916221:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584540924.916223:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540924.916838:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540924.928900:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540919/real 1584540919] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541008 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540924.928909:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540924.928910:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540924.928912:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540924.928919:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540924.928922:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540924.928923:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540924.928924:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540924.928925:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540924.928927:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22722 00000100:00080000:10.0:1584540924.928929:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584540924.932990:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584540924.932994:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584540924.932996:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584540924.932997:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584540924.932998:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584540924.933000:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584540924.933001:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584540924.933003:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541008 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584540924.933033:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078458944 new x1661492078458976 00000800:00020000:2.0:1584540930.916097:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584540930.916104:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584540930.916109:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540930.916629:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540930.928723:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540924/real 1584540924] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541011 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540930.928743:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540930.928744:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540930.928746:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540930.928753:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540930.928755:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540930.928766:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540930.928767:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540930.928768:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540930.928770:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540930.928771:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540930.928772:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22730 00000100:00080000:10.0:1584540930.928774:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584540930.928781:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540930.928783:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540930.928784:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540930.928785:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540930.928786:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540930.928788:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540930.928788:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540930.928790:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540930.932701:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584540930.932705:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584540930.932707:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584540930.932708:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584540930.932709:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584540930.932711:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584540930.932713:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584540930.932714:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541011 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584540930.932743:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078458976 new x1661492078459024 00000800:00020000:2.0:1584540936.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584540936.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584540936.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540936.916522:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540936.928571:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540930/real 1584540930] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541017 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540936.928579:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540936.928580:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540936.928582:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540936.928588:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540936.928590:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540936.928590:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540936.928592:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540936.928592:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540936.928594:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540936.928594:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540936.928596:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22736 00000100:00080000:10.0:1584540936.928599:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584540936.928610:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540936.928611:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540936.928612:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540936.928614:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540936.928614:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540936.928615:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540936.928616:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540936.928617:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584540936.932671:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:8.0:1584540936.932675:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:8.0:1584540936.932677:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:8.0:1584540936.932678:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:8.0:1584540936.932680:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:8.0:1584540936.932682:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:8.0:1584540936.932684:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:8.0:1584540936.932685:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541017 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584540936.932694:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078459024 new x1661492078459072 00000800:00000100:0.0:1584540943.144342:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540943.144782:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540943.156850:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540936/real 1584540936] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541023 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540943.156858:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540943.156859:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540943.156860:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540943.156874:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540943.156876:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540943.156877:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540943.156878:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540943.156879:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540943.156880:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540943.156881:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540943.156883:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22742 00000100:00080000:10.0:1584540943.156885:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584540943.156891:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540943.156892:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540943.156893:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540943.156894:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540943.156895:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540943.156896:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540943.156897:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540943.156898:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540943.160852:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584540943.160857:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584540943.160859:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584540943.160860:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584540943.160861:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584540943.160863:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584540943.160865:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584540943.160867:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541023 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584540943.160895:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078459072 new x1661492078459120 00000800:00020000:4.0:1584540948.916103:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584540948.916106:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584540948.916108:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540948.916497:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540948.928556:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540943/real 1584540943] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541030 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540948.928564:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540948.928566:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540948.928567:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540948.928573:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540948.928574:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540948.928575:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540948.928576:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540948.928578:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540948.928579:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540948.928580:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540948.928581:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22749 00000100:00080000:10.0:1584540948.928584:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584540948.928592:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540948.928594:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540948.928594:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540948.928596:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540948.928596:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540948.928597:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540948.928598:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540948.928599:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540948.932479:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584540948.932484:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584540948.932485:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584540948.932486:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584540948.932487:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584540948.932489:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584540948.932491:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584540948.932492:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541030 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584540948.932500:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078459120 new x1661492078459168 00000800:00020000:2.0:1584540954.916060:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584540954.916063:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584540954.916064:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540954.916557:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540954.928610:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540948/real 1584540948] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541035 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540954.928618:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540954.928619:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540954.928620:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540954.928626:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540954.928627:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540954.928639:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540954.928641:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540954.928641:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540954.928643:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540954.928643:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540954.928645:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22754 00000100:00080000:10.0:1584540954.928647:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584540954.928652:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540954.928654:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540954.928654:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540954.928655:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540954.928656:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540954.928657:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540954.928658:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540954.928659:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540954.932670:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584540954.932674:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584540954.932675:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584540954.932676:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584540954.932677:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584540954.932679:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584540954.932680:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584540954.932682:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541035 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584540954.932711:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078459168 new x1661492078459216 00000800:00020000:6.0:1584540960.916146:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584540960.916148:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584540960.916150:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540960.916567:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540960.928622:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540954/real 1584540954] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541041 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540960.928630:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540960.928631:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540960.928633:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540960.928639:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540960.928641:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540960.928642:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540960.928643:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540960.928644:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540960.928645:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540960.928646:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540960.928647:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22760 00000100:00080000:10.0:1584540960.928650:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584540960.928659:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540960.928661:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540960.928661:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540960.928663:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540960.928663:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540960.928664:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540960.928665:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540960.928666:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540960.932480:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584540960.932484:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584540960.932486:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584540960.932487:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584540960.932489:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584540960.932491:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584540960.932492:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584540960.932494:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541041 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584540960.932503:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078459216 new x1661492078459248 00000800:00000100:0.0:1584540967.144082:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540967.144527:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540967.156582:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540960/real 1584540960] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541047 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540967.156591:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540967.156593:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540967.156595:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540967.156609:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540967.156611:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540967.156612:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540967.156613:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540967.156614:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540967.156615:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540967.156616:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540967.156618:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22766 00000100:00080000:10.0:1584540967.156621:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584540967.156628:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540967.156629:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540967.156630:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540967.156631:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540967.156632:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540967.156633:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540967.156634:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540967.156635:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540967.178078:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584540967.178084:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584540967.178086:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584540967.178087:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584540967.178089:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584540967.178091:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584540967.178093:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584540967.178094:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541047 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584540967.178124:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078459248 new x1661492078459312 00000800:00020000:2.0:1584540972.916056:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584540972.916060:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584540972.916062:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540972.916495:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540972.928577:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540967/real 1584540967] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541054 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540972.928588:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540972.928589:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540972.928591:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540972.928599:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540972.928601:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540972.928602:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540972.928603:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540972.928604:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540972.928605:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540972.928606:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540972.928608:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22773 00000100:00080000:10.0:1584540972.928611:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584540972.928626:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540972.928628:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540972.928629:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540972.928630:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540972.928631:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540972.928632:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540972.928633:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540972.928634:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540972.947251:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584540972.947258:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584540972.947260:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584540972.947261:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584540972.947263:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584540972.947265:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584540972.947267:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584540972.947269:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541054 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584540972.947297:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078459312 new x1661492078459344 00000800:00020000:4.0:1584540978.916048:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584540978.916051:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584540978.916053:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540978.916506:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540978.928560:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540972/real 1584540972] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541059 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540978.928569:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540978.928571:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540978.928572:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540978.928579:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540978.928581:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540978.928594:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540978.928595:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540978.928596:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540978.928598:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540978.928599:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540978.928601:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22778 00000100:00080000:10.0:1584540978.928603:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584540978.928610:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540978.928612:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540978.928613:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540978.928614:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540978.928615:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540978.928616:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540978.928617:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540978.928619:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540978.935117:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584540978.935122:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584540978.935126:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584540978.935127:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584540978.935128:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584540978.935131:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584540978.935133:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584540978.935134:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541059 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584540978.935164:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078459344 new x1661492078459408 00000800:00020000:2.0:1584540984.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584540984.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584540984.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540984.916492:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540984.928543:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540978/real 1584540978] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541065 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540984.928551:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540984.928552:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540984.928553:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540984.928559:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540984.928560:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540984.928561:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540984.928562:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540984.928563:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540984.928564:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540984.928565:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540984.928566:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22784 00000100:00080000:10.0:1584540984.928569:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584540984.928577:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540984.928579:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540984.928579:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540984.928580:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540984.928582:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540984.928583:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540984.928584:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540984.928585:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540984.956607:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584540984.956611:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584540984.956612:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584540984.956613:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584540984.956614:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584540984.956616:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584540984.956618:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584540984.956619:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541065 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584540984.956649:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078459408 new x1661492078459440 00000800:00000100:0.0:1584540991.138938:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584540991.139424:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540991.151480:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540984/real 1584540984] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541071 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540991.151487:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540991.151489:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540991.151490:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540991.151503:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540991.151505:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540991.151506:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540991.151507:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540991.151508:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540991.151509:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540991.151509:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540991.151511:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22790 00000100:00080000:10.0:1584540991.151513:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584540991.151518:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540991.151519:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540991.151521:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540991.151522:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540991.151523:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540991.151524:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540991.151525:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540991.151526:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540991.169727:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584540991.169731:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584540991.169733:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584540991.169733:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584540991.169735:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584540991.169736:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584540991.169738:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584540991.169740:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541071 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584540991.169769:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078459440 new x1661492078459488 00000800:00020000:4.0:1584540996.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584540996.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584540996.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584540996.916530:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584540996.928574:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540991/real 1584540991] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541078 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584540996.928581:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584540996.928582:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584540996.928584:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584540996.928588:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540996.928590:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540996.928591:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540996.928592:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540996.928593:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540996.928594:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584540996.928594:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584540996.928596:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22797 00000100:00080000:10.0:1584540996.928598:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584540996.928608:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540996.928609:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540996.928610:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540996.928611:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540996.928611:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540996.928613:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584540996.928613:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584540996.928614:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584540996.972497:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584540996.972501:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584540996.972503:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584540996.972504:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584540996.972505:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584540996.972507:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584540996.972509:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584540996.972510:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541078 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584540996.972540:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078459488 new x1661492078459536 00000800:00020000:2.0:1584541002.916068:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541002.916071:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541002.916073:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541002.916486:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584541002.928537:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584540996/real 1584540996] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541083 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584541002.928545:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584541002.928546:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584541002.928547:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584541002.928553:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541002.928554:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541002.928562:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541002.928563:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541002.928564:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541002.928566:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584541002.928566:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584541002.928569:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22802 00000100:00080000:10.0:1584541002.928571:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584541002.928576:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541002.928577:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541002.928578:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541002.928579:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541002.928580:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541002.928581:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541002.928581:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541002.928583:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541002.980911:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584541002.980914:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584541002.980916:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584541002.980917:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584541002.980918:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584541002.980920:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584541002.980921:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584541002.980923:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541083 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584541002.980952:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078459536 new x1661492078459584 00000800:00020000:4.0:1584541008.916054:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584541008.916058:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584541008.916059:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541008.916549:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584541008.928601:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541002/real 1584541002] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541089 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584541008.928609:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584541008.928610:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584541008.928611:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584541008.928617:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541008.928618:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541008.928619:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541008.928620:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541008.928621:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541008.928623:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584541008.928624:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584541008.928625:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22808 00000100:00080000:10.0:1584541008.928627:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584541008.928636:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541008.928637:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541008.928638:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541008.928639:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541008.928639:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541008.928640:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541008.928641:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541008.928642:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541008.982569:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584541008.982573:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584541008.982575:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584541008.982576:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584541008.982577:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584541008.982579:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584541008.982580:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584541008.982582:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541089 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584541008.982611:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078459584 new x1661492078459632 00000800:00000100:0.0:1584541015.138360:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541015.138774:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584541015.150827:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541008/real 1584541008] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541095 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584541015.150835:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584541015.150836:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584541015.150837:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584541015.150851:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541015.150853:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541015.150853:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541015.150855:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541015.150856:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541015.150857:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584541015.150858:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584541015.150859:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22814 00000100:00080000:10.0:1584541015.150861:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584541015.150867:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541015.150868:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541015.150869:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541015.150870:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541015.150870:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541015.150872:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541015.150872:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541015.150873:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541015.167548:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584541015.167552:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584541015.167554:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584541015.167555:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584541015.167556:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584541015.167558:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584541015.167560:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584541015.167562:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541095 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584541015.167590:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078459632 new x1661492078459680 00000800:00020000:4.0:1584541020.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584541020.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584541020.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541020.916597:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584541020.928641:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541015/real 1584541015] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541102 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584541020.928649:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584541020.928650:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584541020.928651:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584541020.928657:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541020.928660:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541020.928660:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541020.928662:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541020.928662:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541020.928664:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584541020.928664:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584541020.928666:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22821 00000100:00080000:10.0:1584541020.928668:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584541020.928677:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541020.928678:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541020.928679:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541020.928680:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541020.928680:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541020.928682:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541020.928682:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541020.928683:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541020.969455:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584541020.969459:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584541020.969461:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584541020.969462:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584541020.969463:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584541020.969465:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584541020.969466:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584541020.969468:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541102 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584541020.969479:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078459680 new x1661492078459712 00000800:00020000:6.0:1584541026.916063:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584541026.916066:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584541026.916068:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541026.916460:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584541026.928518:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541020/real 1584541020] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541107 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584541026.928529:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584541026.928530:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584541026.928533:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584541026.928542:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541026.928544:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541026.928557:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541026.928559:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541026.928559:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541026.928561:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584541026.928562:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584541026.928564:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22826 00000100:00080000:10.0:1584541026.928568:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584541026.928575:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541026.928577:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541026.928578:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541026.928579:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541026.928580:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541026.928581:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541026.928582:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541026.928583:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541026.979230:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584541026.979236:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584541026.979238:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584541026.979239:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584541026.979241:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584541026.979244:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584541026.979245:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584541026.979247:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541107 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584541026.979277:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078459712 new x1661492078459776 00000800:00020000:2.0:1584541032.916064:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541032.916067:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541032.916070:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541032.916755:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584541032.928844:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541026/real 1584541026] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541113 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584541032.928859:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584541032.928861:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584541032.928862:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584541032.928870:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541032.928872:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541032.928873:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541032.928874:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541032.928875:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541032.928877:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584541032.928878:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584541032.928880:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22832 00000100:00080000:10.0:1584541032.928883:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584541032.928899:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541032.928901:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541032.928902:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541032.928903:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541032.928903:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541032.928905:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541032.928905:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541032.928907:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541032.970095:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584541032.970103:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584541032.970106:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584541032.970108:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584541032.970110:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584541032.970112:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584541032.970114:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584541032.970116:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541113 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584541032.970147:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078459776 new x1661492078459808 00000800:00000100:0.0:1584541039.141971:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541039.142405:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584541039.154457:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541032/real 1584541032] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541119 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584541039.154465:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584541039.154466:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584541039.154468:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584541039.154482:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541039.154484:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541039.154485:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541039.154486:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541039.154487:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541039.154488:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584541039.154489:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584541039.154491:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22838 00000100:00080000:10.0:1584541039.154494:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584541039.154500:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541039.154502:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541039.154503:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541039.154504:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541039.154504:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541039.154506:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541039.154507:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541039.154508:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541039.175648:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584541039.175653:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584541039.175655:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584541039.175656:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584541039.175658:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584541039.175660:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584541039.175662:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584541039.175664:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541119 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584541039.175675:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078459808 new x1661492078459872 00000800:00020000:0.0:1584541044.916067:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584541044.916069:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584541044.916071:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541044.916517:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584541044.928566:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541039/real 1584541039] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541126 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584541044.928573:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584541044.928574:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584541044.928576:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584541044.928581:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541044.928583:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541044.928584:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541044.928586:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541044.928586:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541044.928588:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584541044.928588:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584541044.928590:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22845 00000100:00080000:14.0:1584541044.928592:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584541044.928602:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541044.928603:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541044.928604:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541044.928605:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541044.928606:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541044.928607:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541044.928608:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541044.928609:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541044.934225:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584541044.934229:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584541044.934231:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584541044.934231:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584541044.934233:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584541044.934234:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584541044.934236:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584541044.934238:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541126 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584541044.934246:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078459872 new x1661492078459904 00000800:00020000:4.0:1584541050.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584541050.916081:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584541050.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541050.916587:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584541050.928680:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541044/real 1584541044] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541131 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584541050.928698:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584541050.928699:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584541050.928700:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584541050.928706:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541050.928707:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541050.928715:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541050.928716:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541050.928717:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541050.928718:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584541050.928719:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584541050.928722:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22850 00000100:00080000:14.0:1584541050.928735:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584541050.928740:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541050.928741:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541050.928742:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541050.928743:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541050.928744:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541050.928745:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541050.928746:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541050.928747:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541050.934293:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584541050.934296:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584541050.934299:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584541050.934300:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584541050.934301:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584541050.934303:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584541050.934305:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584541050.934306:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541131 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584541050.934318:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078459904 new x1661492078459952 00000800:00020000:0.0:1584541056.916096:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584541056.916098:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584541056.916100:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541056.916549:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584541056.928599:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541050/real 1584541050] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541137 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584541056.928608:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584541056.928609:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584541056.928610:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584541056.928616:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541056.928618:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541056.928619:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541056.928620:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541056.928621:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541056.928622:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584541056.928623:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584541056.928625:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22856 00000100:00080000:16.0:1584541056.928628:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584541056.928637:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541056.928639:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541056.928639:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541056.928641:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541056.928641:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541056.928642:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541056.928643:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541056.928644:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541056.979192:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584541056.979196:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584541056.979197:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584541056.979198:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584541056.979200:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584541056.979202:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584541056.979204:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584541056.979205:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541137 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584541056.979217:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078459952 new x1661492078460000 00000800:00000100:0.0:1584541063.142665:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541063.143077:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584541063.155130:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541056/real 1584541056] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541143 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584541063.155138:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584541063.155139:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584541063.155141:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584541063.155154:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541063.155156:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541063.155157:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541063.155158:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541063.155159:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541063.155161:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584541063.155161:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584541063.155163:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22862 00000100:00080000:16.0:1584541063.155165:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584541063.155170:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541063.155171:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541063.155172:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541063.155173:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541063.155174:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541063.155175:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541063.155176:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541063.155177:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541063.190866:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584541063.190869:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584541063.190871:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584541063.190872:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584541063.190873:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584541063.190875:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584541063.190876:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584541063.190878:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541143 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584541063.190907:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078460000 new x1661492078460048 00000800:00020000:4.0:1584541068.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584541068.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584541068.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541068.916548:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584541068.928630:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541063/real 1584541063] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541150 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584541068.928638:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584541068.928639:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584541068.928641:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584541068.928647:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541068.928648:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541068.928649:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541068.928651:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541068.928652:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541068.928653:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584541068.928654:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584541068.928655:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22869 00000100:00080000:16.0:1584541068.928657:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584541068.928666:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541068.928667:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541068.928668:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541068.928669:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541068.928669:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541068.928671:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541068.928671:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541068.928672:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541068.966669:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584541068.966673:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584541068.966675:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584541068.966676:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584541068.966677:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584541068.966679:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584541068.966681:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584541068.966682:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541150 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584541068.966712:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078460048 new x1661492078460096 00000800:00020000:2.0:1584541074.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541074.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541074.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541074.916716:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584541074.928806:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541068/real 1584541068] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541155 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584541074.928814:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584541074.928815:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584541074.928816:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584541074.928822:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541074.928825:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541074.928833:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541074.928834:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541074.928834:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541074.928836:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584541074.928836:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584541074.928838:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22874 00000100:00080000:16.0:1584541074.928851:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584541074.928856:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541074.928857:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541074.928858:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541074.928870:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541074.928870:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541074.928872:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541074.928872:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541074.928873:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541074.969955:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584541074.969959:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584541074.969961:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584541074.969961:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584541074.969963:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584541074.969965:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584541074.969966:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584541074.969968:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541155 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584541074.969999:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078460096 new x1661492078460144 00000800:00020000:2.0:1584541080.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541080.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541080.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541080.916496:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584541080.928550:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541074/real 1584541074] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541161 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584541080.928558:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584541080.928560:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584541080.928562:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584541080.928568:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541080.928570:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541080.928571:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541080.928572:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541080.928572:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541080.928574:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584541080.928574:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584541080.928576:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22880 00000100:00080000:16.0:1584541080.928578:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584541080.928587:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541080.928588:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541080.928589:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541080.928590:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541080.928591:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541080.928592:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541080.928592:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541080.928593:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541080.976640:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584541080.976643:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584541080.976645:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584541080.976646:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584541080.976647:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584541080.976649:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584541080.976650:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584541080.976652:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541161 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584541080.976681:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078460144 new x1661492078460176 00000800:00000100:0.0:1584541087.143071:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541087.143531:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584541087.155600:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541080/real 1584541080] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541167 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584541087.155614:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584541087.155615:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584541087.155617:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584541087.155636:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541087.155638:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541087.155639:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541087.155640:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541087.155652:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541087.155653:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584541087.155654:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584541087.155657:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22886 00000100:00080000:16.0:1584541087.155659:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584541087.155678:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541087.155680:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541087.155681:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541087.155682:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541087.155683:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541087.155684:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541087.155685:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541087.155697:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541087.211812:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584541087.211819:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584541087.211822:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584541087.211823:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584541087.211825:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584541087.211827:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584541087.211829:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584541087.211831:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541167 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584541087.211860:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078460176 new x1661492078460240 00000800:00020000:2.0:1584541092.916070:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541092.916073:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541092.916075:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541092.916520:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584541092.928642:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541087/real 1584541087] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541174 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584541092.928652:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584541092.928654:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584541092.928656:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584541092.928664:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541092.928666:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541092.928667:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541092.928668:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541092.928669:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541092.928671:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584541092.928672:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584541092.928674:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22893 00000100:00080000:16.0:1584541092.928677:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584541092.928693:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541092.928694:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541092.928695:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541092.928696:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541092.928697:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541092.928698:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541092.928700:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541092.928702:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541092.961306:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584541092.961317:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584541092.961321:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584541092.961323:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584541092.961327:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584541092.961331:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584541092.961335:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584541092.961339:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541174 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584541092.961401:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078460240 new x1661492078460272 00000800:00020000:2.0:1584541098.916056:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541098.916064:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541098.916068:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541098.916579:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584541098.928664:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541092/real 1584541092] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541179 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584541098.928685:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584541098.928686:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584541098.928688:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584541098.928695:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541098.928697:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541098.928710:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541098.928711:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541098.928712:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541098.928713:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584541098.928714:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584541098.928727:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22898 00000100:00080000:16.0:1584541098.928730:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584541098.928737:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541098.928738:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541098.928739:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541098.928741:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541098.928742:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541098.928744:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541098.928744:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541098.928746:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541098.934638:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541098.934644:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541098.934647:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541098.934648:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541098.934649:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541098.934652:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541098.934653:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541098.934655:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541179 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541098.934686:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078460272 new x1661492078460336 00000800:00020000:2.0:1584541104.916059:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541104.916062:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541104.916064:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541104.916703:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541104.928759:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541098/real 1584541098] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541185 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541104.928768:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541104.928769:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541104.928771:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541104.928778:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541104.928780:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541104.928781:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541104.928782:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541104.928783:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541104.928784:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541104.928785:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541104.928787:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22904 00000100:00080000:18.0:1584541104.928789:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541104.928803:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541104.928804:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541104.928805:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541104.928806:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541104.928808:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541104.928810:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541104.928810:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541104.928812:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541104.984950:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541104.984956:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541104.984958:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541104.984959:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541104.984960:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541104.984963:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541104.984964:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541104.984966:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541185 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541104.984996:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078460336 new x1661492078460368 00000800:00000100:0.0:1584541111.139670:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541111.140098:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541111.152165:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541104/real 1584541104] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541191 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541111.152172:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541111.152173:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541111.152174:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541111.152188:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541111.152190:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541111.152190:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541111.152191:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541111.152192:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541111.152193:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541111.152205:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541111.152207:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22910 00000100:00080000:18.0:1584541111.152209:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541111.152226:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541111.152227:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541111.152229:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541111.152230:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541111.152231:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541111.152232:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541111.152233:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541111.152234:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541111.184377:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541111.184385:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541111.184389:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541111.184391:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541111.184394:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541111.184400:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541111.184403:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541111.184407:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541191 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541111.184459:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078460368 new x1661492078460416 00000800:00020000:2.0:1584541116.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541116.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541116.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541116.916622:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541116.928706:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541111/real 1584541111] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541198 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541116.928715:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541116.928716:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541116.928717:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541116.928723:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541116.928725:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541116.928726:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541116.928727:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541116.928728:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541116.928729:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541116.928730:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541116.928731:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22917 00000100:00080000:18.0:1584541116.928733:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541116.928744:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541116.928746:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541116.928747:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541116.928748:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541116.928748:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541116.928750:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541116.928750:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541116.928751:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541116.973531:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541116.973536:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541116.973537:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541116.973538:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541116.973540:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541116.973542:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541116.973543:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541116.973545:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541198 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541116.973576:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078460416 new x1661492078460464 00000800:00020000:2.0:1584541122.916091:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541122.916093:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541122.916095:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541122.916554:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541122.928639:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541116/real 1584541116] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541203 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541122.928658:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541122.928659:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541122.928660:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541122.928665:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541122.928667:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541122.928675:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541122.928676:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541122.928677:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541122.928679:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541122.928679:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541122.928693:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22922 00000100:00080000:18.0:1584541122.928695:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541122.928700:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541122.928701:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541122.928702:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541122.928703:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541122.928703:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541122.928705:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541122.928705:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541122.928706:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541122.976645:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541122.976649:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541122.976651:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541122.976651:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541122.976653:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541122.976654:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541122.976656:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541122.976658:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541203 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541122.976686:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078460464 new x1661492078460512 00000800:00020000:4.0:1584541128.916069:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584541128.916072:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584541128.916074:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541128.916490:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541128.928566:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541122/real 1584541122] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541209 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541128.928574:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541128.928575:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541128.928576:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541128.928582:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541128.928584:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541128.928585:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541128.928586:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541128.928587:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541128.928589:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541128.928589:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541128.928591:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22928 00000100:00080000:18.0:1584541128.928593:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541128.928602:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541128.928603:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541128.928604:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541128.928605:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541128.928605:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541128.928606:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541128.928607:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541128.928608:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541128.968112:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541128.968117:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541128.968118:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541128.968119:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541128.968120:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541128.968123:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541128.968125:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541128.968126:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541209 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541128.968155:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078460512 new x1661492078460560 00000800:00000100:0.0:1584541135.139760:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541135.140267:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541135.152338:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541128/real 1584541128] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541215 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541135.152345:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541135.152346:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541135.152347:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541135.152360:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541135.152362:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541135.152362:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541135.152364:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541135.152365:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541135.152377:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541135.152378:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541135.152379:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22934 00000100:00080000:18.0:1584541135.152381:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541135.152397:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541135.152398:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541135.152399:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541135.152400:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541135.152401:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541135.152402:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541135.152403:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541135.152404:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541135.213004:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541135.213007:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541135.213009:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541135.213010:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541135.213011:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541135.213013:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541135.213014:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541135.213016:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541215 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541135.213050:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078460560 new x1661492078460608 00000800:00020000:4.0:1584541140.916064:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584541140.916067:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584541140.916069:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541140.916450:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541140.928530:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541135/real 1584541135] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541222 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541140.928538:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541140.928540:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541140.928541:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541140.928547:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541140.928550:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541140.928551:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541140.928552:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541140.928553:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541140.928554:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541140.928555:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541140.928557:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22941 00000100:00080000:18.0:1584541140.928559:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541140.928569:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541140.928571:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541140.928571:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541140.928572:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541140.928573:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541140.928574:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541140.928575:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541140.928576:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541140.972405:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541140.972410:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541140.972411:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541140.972412:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541140.972414:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541140.972416:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541140.972417:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541140.972419:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541222 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541140.972449:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078460608 new x1661492078460640 00000800:00020000:2.0:1584541146.916088:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541146.916093:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541146.916098:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541146.916649:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541146.928739:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541140/real 1584541140] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541227 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541146.928747:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541146.928749:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541146.928751:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541146.928757:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541146.928759:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541146.928769:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541146.928771:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541146.928771:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541146.928773:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541146.928773:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541146.928775:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22946 00000100:00080000:18.0:1584541146.928777:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541146.928783:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541146.928784:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541146.928785:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541146.928786:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541146.928787:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541146.928788:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541146.928788:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541146.928790:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541146.978664:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541146.978673:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541146.978677:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541146.978679:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541146.978682:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541146.978688:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541146.978692:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541146.978696:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541227 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541146.978744:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078460640 new x1661492078460704 00000800:00020000:2.0:1584541152.916067:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541152.938884:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541152.963059:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541152.963509:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541152.975588:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541146/real 1584541146] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541233 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541152.975603:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541152.975605:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541152.975606:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541152.975614:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541152.975617:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541152.975617:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541152.975619:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541152.975620:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541152.975621:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541152.975622:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541152.975624:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22952 00000100:00080000:18.0:1584541152.975627:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541152.975642:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541152.975644:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541152.975644:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541152.975645:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541152.975646:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541152.975647:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541152.975648:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541152.975649:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541152.993092:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541152.993098:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541152.993100:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541152.993101:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541152.993103:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541152.993105:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541152.993107:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541152.993109:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541233 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541152.993138:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078460704 new x1661492078460736 00000800:00000100:0.0:1584541159.141258:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541159.141762:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541159.153849:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541152/real 1584541152] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541239 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541159.153859:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541159.153861:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541159.153863:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541159.153886:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541159.153899:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541159.153900:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541159.153901:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541159.153902:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541159.153904:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541159.153905:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541159.153918:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22958 00000100:00080000:18.0:1584541159.153922:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541159.153929:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541159.153931:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541159.153937:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541159.153938:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541159.153939:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541159.153940:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541159.153942:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541159.153943:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541159.217514:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541159.217521:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541159.217523:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541159.217524:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541159.217526:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541159.217529:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541159.217530:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541159.217532:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541239 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541159.217565:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078460736 new x1661492078460800 00000800:00020000:4.0:1584541164.916042:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584541164.916045:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584541164.916047:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541164.916429:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541164.928484:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541159/real 1584541159] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541246 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541164.928493:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541164.928494:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541164.928496:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541164.928503:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541164.928504:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541164.928505:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541164.928507:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541164.928507:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541164.928509:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541164.928510:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541164.928512:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22965 00000100:00080000:18.0:1584541164.928514:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541164.928527:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541164.928529:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541164.928530:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541164.928531:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541164.928533:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541164.928534:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541164.928535:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541164.928536:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541164.949427:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541164.949433:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541164.949435:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541164.949436:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541164.949438:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541164.949441:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541164.949443:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541164.949445:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541246 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541164.949475:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078460800 new x1661492078460832 00000800:00020000:0.0:1584541170.916059:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584541170.916062:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584541170.916063:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541170.916499:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541170.928576:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541164/real 1584541164] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541251 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541170.928584:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541170.928585:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541170.928586:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541170.928592:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541170.928594:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541170.928605:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541170.928607:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541170.928607:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541170.928609:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541170.928609:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541170.928611:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22970 00000100:00080000:18.0:1584541170.928613:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541170.928618:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541170.928620:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541170.928621:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541170.928623:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541170.928623:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541170.928624:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541170.928625:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541170.928626:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541170.934447:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541170.934451:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541170.934453:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541170.934454:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541170.934455:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541170.934457:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541170.934459:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541170.934461:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541251 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541170.934490:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078460832 new x1661492078460880 00000800:00020000:2.0:1584541176.916052:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541176.916054:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541176.916056:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541176.916463:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541176.928517:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541170/real 1584541170] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541257 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541176.928526:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541176.928527:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541176.928528:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541176.928533:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541176.928535:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541176.928536:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541176.928537:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541176.928538:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541176.928539:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541176.928540:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541176.928541:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22976 00000100:00080000:18.0:1584541176.928544:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541176.928553:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541176.928555:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541176.928555:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541176.928556:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541176.928557:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541176.928558:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541176.928559:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541176.928560:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541176.981882:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541176.981887:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541176.981888:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541176.981889:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541176.981890:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541176.981892:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541176.981894:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541176.981896:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541257 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541176.981925:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078460880 new x1661492078460928 00000800:00000100:0.0:1584541183.141704:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541183.142134:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541183.154202:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541176/real 1584541176] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541263 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541183.154210:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541183.154211:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541183.154212:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541183.154226:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541183.154227:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541183.154228:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541183.154229:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541183.154230:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541183.154231:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541183.154232:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541183.154234:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22982 00000100:00080000:18.0:1584541183.154236:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541183.154242:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541183.154244:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541183.154244:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541183.154245:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541183.154246:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541183.154248:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541183.154248:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541183.154249:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541183.193095:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541183.193099:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541183.193100:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541183.193101:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541183.193102:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541183.193105:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541183.193107:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541183.193108:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541263 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584541183.193138:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078460928 new x1661492078460976 00000800:00020000:2.0:1584541188.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541188.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541188.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541188.916620:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584541188.928672:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541183/real 1584541183] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541270 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584541188.928680:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584541188.928681:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584541188.928683:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584541188.928689:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541188.928690:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541188.928691:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541188.928692:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541188.928693:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541188.928694:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584541188.928695:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584541188.928697:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22989 00000100:00080000:22.0:1584541188.928699:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584541188.928708:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541188.928709:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541188.928710:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541188.928711:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541188.928712:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541188.928714:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541188.928714:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541188.928715:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541188.961640:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541188.961644:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541188.961646:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541188.961647:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541188.961648:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541188.961650:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541188.961652:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541188.961653:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541270 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584541188.961662:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078460976 new x1661492078461024 00000800:00020000:2.0:1584541194.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541194.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541194.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541194.916520:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584541194.928572:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541188/real 1584541188] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584541194.928580:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584541194.928581:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584541194.928583:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584541194.928588:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541194.928590:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541194.928599:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541194.928600:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541194.928600:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541194.928602:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584541194.928602:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584541194.928604:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22994 00000100:00080000:22.0:1584541194.928606:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584541194.928612:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541194.928613:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541194.928615:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541194.928616:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541194.928616:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541194.928617:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541194.928618:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541194.928619:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541194.976642:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541194.976646:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541194.976647:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541194.976648:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541194.976649:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541194.976651:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541194.976652:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541194.976654:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584541194.976684:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078461024 new x1661492078461072 00000800:00020000:2.0:1584541200.916052:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541200.916056:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541200.916058:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541200.916526:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584541200.928584:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541194/real 1584541194] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584541200.928596:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584541200.928597:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584541200.928599:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584541200.928607:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541200.928609:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541200.928610:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541200.928611:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541200.928612:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541200.928613:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584541200.928615:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584541200.928617:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23000 00000100:00080000:6.0:1584541200.928621:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584541200.928639:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541200.928640:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541200.928641:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541200.928642:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541200.928643:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541200.928644:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541200.928646:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541200.928648:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541200.961013:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541200.961021:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541200.961023:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541200.961024:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541200.961038:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541200.961041:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541200.961043:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541200.961045:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584541200.961054:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078461072 new x1661492078461104 00000800:00000100:0.0:1584541207.142092:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541207.142524:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584541207.154606:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541200/real 1584541200] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584541207.154616:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584541207.154618:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584541207.154620:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584541207.154641:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541207.154643:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541207.154644:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541207.154645:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541207.154646:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541207.154648:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584541207.154649:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584541207.154651:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23006 00000100:00080000:6.0:1584541207.154654:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584541207.154661:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541207.154663:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541207.154664:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541207.154665:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541207.154667:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541207.154668:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541207.154669:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541207.154670:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000800:00000400:2.0:1584541212.916058:0:25711:0:(o2iblnd_cb.c:3381:kiblnd_check_conns()) Timed out tx for 10.10.2.22@o2ib: 0 seconds 00000400:00000100:2.0:1584541212.927236:0:25711:0:(lib-msg.c:705:lnet_health_check()) msg 0@<0:0>->10.10.2.22@o2ib exceeded retry count 0 00000100:00080000:20.0:1584541212.927268:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:20.0:1584541212.927271:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0003_UUID on 10.10.2.22@o2ib failed (-110) 00000100:00080000:6.0:1584541232.217077:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541232.217079:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541232.217096:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541232.217097:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541232.217098:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:6.0:1584541232.217100:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584541232.217101:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584541232.217105:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23013 00000100:00080000:6.0:1584541232.217106:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 20776 00000100:00080000:6.0:1584541232.217110:0:42:0:(import.c:596:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0003 (at 10.10.2.21@o2ib) 00000100:00080000:6.0:1584541232.217111:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.21@o2ib/10.10.2.21@o2ib 00000100:00080000:6.0:1584541232.217119:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541232.217120:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541232.217121:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541232.217122:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541232.217123:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541232.217126:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541232.217127:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541232.217128:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541232.217330:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:20.0:1584541232.217333:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0003_UUID on 10.10.2.21@o2ib failed (-19) 00000100:00080000:6.0:1584541257.241061:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541257.241063:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541257.241069:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541257.241070:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541257.241071:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:6.0:1584541257.241073:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584541257.241074:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584541257.241076:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23013 00000100:00080000:6.0:1584541257.241078:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 23038 00000100:00080000:6.0:1584541257.241079:0:42:0:(import.c:570:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: tried all connections, increasing latency to 6s 00000100:00080000:6.0:1584541257.241081:0:42:0:(import.c:596:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0003 (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584541257.241082:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584541257.241089:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541257.241091:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541257.241091:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541257.241093:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541257.241094:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541257.241095:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541257.241096:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541257.241098:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541257.241242:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541257.241248:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541257.241251:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541257.241252:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541257.241254:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541257.241257:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541257.241258:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541257.241261:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584541257.241272:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078461104 new x1661492078461312 00000800:00020000:4.0:1584541262.916063:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584541262.916067:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584541262.916069:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541262.916488:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584541262.928640:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541257/real 1584541257] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541344 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584541262.928651:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584541262.928652:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584541262.928654:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584541262.928662:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541262.928664:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541262.928666:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541262.928667:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541262.928667:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541262.928669:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584541262.928670:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584541262.928673:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23063 00000100:00080000:8.0:1584541262.928676:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584541262.928695:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541262.928696:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541262.928697:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541262.928699:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541262.928700:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541262.928701:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541262.928702:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541262.928703:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541262.934522:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541262.934530:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541262.934532:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541262.934534:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541262.934536:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541262.934538:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541262.934540:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541262.934542:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541344 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584541262.934556:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078461312 new x1661492078461360 00000800:00020000:4.0:1584541268.916062:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584541268.916065:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584541268.916067:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541268.916475:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584541268.928561:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541262/real 1584541262] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541349 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584541268.928571:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584541268.928572:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584541268.928574:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584541268.928581:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541268.928583:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541268.928583:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541268.928585:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541268.928586:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541268.928587:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584541268.928588:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584541268.928590:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23068 00000100:00080000:8.0:1584541268.928592:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584541268.928605:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541268.928608:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541268.928608:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541268.928610:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541268.928610:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541268.928612:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541268.928612:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541268.928613:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541268.934284:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541268.934290:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541268.934292:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541268.934293:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541268.934295:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541268.934297:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541268.934299:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541268.934301:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541349 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584541268.934310:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078461360 new x1661492078461392 00000800:00020000:2.0:1584541274.916069:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541274.916072:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541274.916074:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541274.916470:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584541274.928594:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541268/real 1584541268] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541355 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584541274.928604:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584541274.928605:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584541274.928607:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584541274.928614:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541274.928616:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541274.928617:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541274.928618:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541274.928619:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541274.928621:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584541274.928621:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584541274.928623:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23074 00000100:00080000:8.0:1584541274.928627:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584541274.928640:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541274.928642:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541274.928642:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541274.928644:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541274.928644:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541274.928645:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541274.928646:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541274.928647:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541274.934579:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541274.934586:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541274.934588:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541274.934589:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541274.934591:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541274.934593:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541274.934595:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541274.934597:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541355 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584541274.934626:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078461392 new x1661492078461424 00000800:00020000:4.0:1584541280.916040:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584541280.916043:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584541280.916046:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541280.916492:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584541280.928567:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541274/real 1584541274] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541361 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584541280.928578:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584541280.928580:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584541280.928582:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584541280.928602:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541280.928604:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541280.928610:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541280.928612:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541280.928612:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541280.928614:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584541280.928616:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584541280.928618:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23080 00000100:00080000:8.0:1584541280.928621:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584541280.928629:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541280.928630:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541280.928631:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541280.928632:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541280.928633:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541280.928634:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541280.928635:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541280.928636:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541280.934533:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541280.934540:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541280.934542:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541280.934544:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541280.934546:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541280.934549:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541280.934550:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541280.934552:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541361 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584541280.934562:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078461424 new x1661492078461504 00000800:00020000:2.0:1584541286.916101:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541286.916106:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541286.916111:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541286.916614:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584541286.928691:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541280/real 1584541280] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541367 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584541286.928709:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584541286.928710:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584541286.928712:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584541286.928717:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541286.928718:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541286.928719:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541286.928720:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541286.928722:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541286.928723:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584541286.928724:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584541286.928725:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23086 00000100:00080000:8.0:1584541286.928727:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584541286.928736:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541286.928737:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541286.928738:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541286.928739:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541286.928739:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541286.928741:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541286.928741:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541286.928742:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541286.934711:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541286.934715:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541286.934716:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541286.934730:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541286.934731:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541286.934733:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541286.934734:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541286.934736:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541367 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584541286.934745:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078461504 new x1661492078461536 00000800:00020000:4.0:1584541292.916084:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584541292.916088:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584541292.916090:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541292.916506:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584541292.928585:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541286/real 1584541286] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541373 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584541292.928594:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584541292.928595:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584541292.928596:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584541292.928603:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541292.928605:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541292.928607:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541292.928608:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541292.928609:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541292.928611:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584541292.928611:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584541292.928613:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23092 00000100:00080000:8.0:1584541292.928616:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584541292.928625:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541292.928627:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541292.928627:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541292.928629:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541292.928629:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541292.928631:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541292.928631:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541292.928632:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541292.960644:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541292.960649:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541292.960651:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541292.960652:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541292.960654:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541292.960656:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541292.960658:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541292.960660:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541373 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584541292.960690:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078461536 new x1661492078461584 00000800:00020000:2.0:1584541298.916096:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541298.916102:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541298.916106:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541298.916627:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584541298.928718:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541292/real 1584541292] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541379 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584541298.928726:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584541298.928738:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584541298.928739:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584541298.928745:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541298.928747:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541298.928748:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541298.928749:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541298.928749:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541298.928751:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584541298.928751:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584541298.928753:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23098 00000100:00080000:8.0:1584541298.928755:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584541298.928763:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541298.928764:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541298.928765:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541298.928766:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541298.928766:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541298.928767:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541298.928768:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541298.928769:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541298.940862:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541298.940871:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541298.940874:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541298.940876:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541298.940879:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541298.940883:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541298.940887:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541298.940891:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541379 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584541298.940923:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078461584 new x1661492078461616 00000800:00020000:4.0:1584541304.916166:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584541304.916169:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584541304.916171:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541304.916604:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584541304.928680:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541298/real 1584541298] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541385 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584541304.928688:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584541304.928690:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584541304.928691:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584541304.928705:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541304.928707:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541304.928712:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541304.928713:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541304.928714:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541304.928715:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584541304.928716:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584541304.928717:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23104 00000100:00080000:8.0:1584541304.928719:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584541304.928724:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541304.928725:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541304.928726:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541304.928727:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541304.928728:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541304.928729:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541304.928730:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541304.928731:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541304.961965:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541304.961969:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541304.961970:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541304.961971:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541304.961972:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541304.961974:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541304.961976:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541304.961977:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541385 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584541304.962006:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078461616 new x1661492078461680 00000800:00020000:4.0:1584541310.916124:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584541310.916130:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584541310.916134:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541310.916652:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584541310.928742:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541304/real 1584541304] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541391 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584541310.928750:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584541310.928751:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584541310.928763:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584541310.928769:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541310.928771:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541310.928772:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541310.928773:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541310.928774:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541310.928775:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584541310.928776:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584541310.928777:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23110 00000100:00080000:8.0:1584541310.928779:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584541310.928788:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541310.928789:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541310.928790:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541310.928791:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541310.928791:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541310.928793:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541310.928793:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541310.928794:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541310.962488:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541310.962492:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541310.962493:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541310.962494:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541310.962496:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541310.962497:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541310.962499:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541310.962501:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541391 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584541310.962510:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078461680 new x1661492078461728 00000800:00020000:2.0:1584541316.916059:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541316.916062:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541316.916064:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541316.916624:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584541316.928706:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541310/real 1584541310] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541397 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584541316.928713:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584541316.928714:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584541316.928716:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584541316.928721:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541316.928723:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541316.928724:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541316.928725:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541316.928725:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541316.928727:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584541316.928727:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584541316.928729:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23116 00000100:00080000:8.0:1584541316.928731:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584541316.928738:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541316.928740:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541316.928740:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541316.928741:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541316.928742:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541316.928743:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541316.928745:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541316.928746:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541316.947688:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541316.947692:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541316.947693:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541316.947694:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541316.947695:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541316.947697:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541316.947699:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541316.947700:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541397 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584541316.947733:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078461728 new x1661492078461760 00000800:00000100:0.0:1584541323.140753:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541323.141197:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584541323.153266:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541316/real 1584541316] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541403 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584541323.153274:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584541323.153275:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584541323.153277:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584541323.153283:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541323.153285:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541323.153286:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541323.153287:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541323.153288:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541323.153289:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584541323.153290:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584541323.153292:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23122 00000100:00080000:8.0:1584541323.153294:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584541323.153304:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541323.153305:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541323.153306:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541323.153307:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541323.153309:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541323.153310:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541323.153310:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541323.153312:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541323.184592:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541323.184597:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541323.184599:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541323.184600:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541323.184601:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541323.184604:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541323.184605:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541323.184607:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541403 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584541323.184636:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078461760 new x1661492078461808 00000800:00020000:2.0:1584541328.916067:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541328.916070:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541328.916072:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541328.916542:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584541328.928618:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541323/real 1584541323] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541410 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584541328.928626:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584541328.928627:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584541328.928629:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584541328.928642:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541328.928644:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541328.928648:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541328.928649:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541328.928650:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541328.928651:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584541328.928652:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584541328.928654:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23129 00000100:00080000:8.0:1584541328.928656:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584541328.928661:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541328.928662:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541328.928664:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541328.928665:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541328.928665:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541328.928667:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584541328.928667:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584541328.928668:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541328.975712:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541328.975716:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541328.975718:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541328.975719:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541328.975720:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541328.975722:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541328.975723:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541328.975725:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541410 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584541328.975755:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078461808 new x1661492078461872 00000800:00020000:2.0:1584541334.916098:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541334.916104:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541334.916109:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541334.916624:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584541334.928712:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541328/real 1584541328] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541415 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584541334.928735:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584541334.928736:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584541334.928739:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584541334.928747:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541334.928749:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541334.928749:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541334.928751:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541334.928751:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541334.928753:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584541334.928754:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584541334.928756:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23134 00000100:00080000:10.0:1584541334.928760:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584541334.928777:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541334.928778:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541334.928779:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541334.928780:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541334.928782:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541334.928783:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541334.928795:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541334.928796:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541334.971051:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541334.971069:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541334.971071:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541334.971073:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541334.971075:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541334.971077:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541334.971079:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541334.971080:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541415 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584541334.971127:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078461872 new x1661492078461904 00000800:00020000:2.0:1584541340.916086:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541340.916089:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541340.916091:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541340.916532:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584541340.928616:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541334/real 1584541334] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541421 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584541340.928626:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584541340.928627:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584541340.928629:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584541340.928637:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541340.928639:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541340.928640:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541340.928641:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541340.928642:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541340.928644:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584541340.928644:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584541340.928646:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23140 00000100:00080000:10.0:1584541340.928649:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584541340.928662:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541340.928664:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541340.928666:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541340.928667:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541340.928668:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541340.928669:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541340.928670:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541340.928671:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541340.987229:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541340.987236:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541340.987238:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541340.987239:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541340.987241:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541340.987244:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541340.987245:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541340.987247:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541421 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584541340.987276:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078461904 new x1661492078461952 00000800:00000100:0.0:1584541347.140502:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541347.140986:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584541347.153062:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541340/real 1584541340] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541427 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584541347.153072:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584541347.153074:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584541347.153075:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584541347.153083:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541347.153085:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541347.153085:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541347.153087:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541347.153087:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541347.153089:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584541347.153090:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584541347.153091:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23146 00000100:00080000:10.0:1584541347.153094:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584541347.153110:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541347.153111:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541347.153112:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541347.153113:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541347.153113:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541347.153115:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541347.153126:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541347.153127:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541347.193119:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541347.193126:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541347.193128:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541347.193129:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541347.193131:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541347.193133:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541347.193135:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541347.193137:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541427 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584541347.193171:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078461952 new x1661492078461984 00000800:00020000:2.0:1584541352.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541352.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541352.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541352.916508:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584541352.928589:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541347/real 1584541347] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541434 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584541352.928597:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584541352.928598:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584541352.928599:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584541352.928613:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541352.928615:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541352.928620:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541352.928622:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541352.928623:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541352.928624:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584541352.928625:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584541352.928628:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23153 00000100:00080000:10.0:1584541352.928630:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584541352.928635:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541352.928636:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541352.928637:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541352.928638:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541352.928639:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541352.928640:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541352.928641:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541352.928642:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541352.969943:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541352.969953:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541352.969956:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541352.969958:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541352.969962:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541352.969966:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541352.969970:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541352.969974:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541434 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584541352.970024:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078461984 new x1661492078462064 00000800:00020000:2.0:1584541358.916066:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541358.916068:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541358.916070:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541358.916456:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584541358.928546:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541352/real 1584541352] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541439 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584541358.928565:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584541358.928566:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584541358.928567:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584541358.928574:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541358.928576:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541358.928576:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541358.928579:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541358.928579:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541358.928581:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584541358.928581:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584541358.928583:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23158 00000100:00080000:12.0:1584541358.928585:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584541358.928596:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541358.928597:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541358.928598:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541358.928599:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541358.928599:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541358.928601:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541358.928602:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541358.928603:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541358.972532:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541358.972537:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541358.972539:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541358.972540:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541358.972541:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541358.972543:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541358.972545:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541358.972547:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541439 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584541358.972577:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078462064 new x1661492078462096 00000800:00020000:2.0:1584541364.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541364.916081:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541364.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541364.916588:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584541364.928640:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541358/real 1584541358] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541445 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584541364.928648:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584541364.928649:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584541364.928651:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584541364.928656:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541364.928659:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541364.928659:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541364.928660:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541364.928661:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541364.928662:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584541364.928663:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584541364.928665:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23164 00000100:00080000:12.0:1584541364.928667:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584541364.928675:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541364.928676:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541364.928677:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541364.928678:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541364.928678:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541364.928680:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541364.928680:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541364.928682:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541364.970893:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541364.970897:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541364.970899:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541364.970899:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541364.970901:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541364.970902:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541364.970904:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541364.970906:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541445 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584541364.970934:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078462096 new x1661492078462128 00000800:00000100:0.0:1584541371.141147:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541371.141681:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584541371.153757:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541364/real 1584541364] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541451 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584541371.153764:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584541371.153765:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584541371.153767:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584541371.153773:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541371.153774:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541371.153775:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541371.153776:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541371.153777:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541371.153778:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584541371.153779:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584541371.153780:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23170 00000100:00080000:12.0:1584541371.153782:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584541371.153790:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541371.153791:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541371.153792:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541371.153793:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541371.153793:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541371.153794:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541371.153795:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541371.153796:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541371.159846:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541371.159851:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541371.159852:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541371.159853:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541371.159854:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541371.159856:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541371.159858:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541371.159859:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541451 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584541371.159894:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078462128 new x1661492078462176 00000800:00020000:2.0:1584541376.916076:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541376.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541376.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541376.916498:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584541376.928551:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541371/real 1584541371] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541458 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584541376.928563:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584541376.928564:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584541376.928566:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584541376.928579:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541376.928581:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541376.928586:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541376.928587:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541376.928588:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541376.928590:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584541376.928590:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584541376.928592:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23177 00000100:00080000:12.0:1584541376.928594:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584541376.928598:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541376.928600:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541376.928600:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541376.928601:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541376.928602:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541376.928603:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541376.928604:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541376.928605:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541376.934664:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541376.934672:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541376.934675:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541376.934677:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541376.934680:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541376.934684:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541376.934688:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541376.934692:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541458 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584541376.934739:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078462176 new x1661492078462240 00000800:00020000:4.0:1584541382.916116:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584541382.916122:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584541382.916127:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541382.916564:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584541382.928759:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541376/real 1584541376] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541463 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584541382.928770:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584541382.928771:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584541382.928773:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584541382.928782:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541382.928784:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541382.928785:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541382.928786:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541382.928787:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541382.928788:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584541382.928789:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584541382.928792:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23182 00000100:00080000:12.0:1584541382.928795:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584541382.928809:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541382.928811:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541382.928812:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541382.928813:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541382.928814:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541382.928815:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541382.928817:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541382.928818:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541382.934684:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541382.934691:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541382.934693:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541382.934694:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541382.934697:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541382.934699:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541382.934701:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541382.934703:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541463 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584541382.934737:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078462240 new x1661492078462288 00000800:00020000:2.0:1584541388.916061:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541388.916064:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541388.916065:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541388.916562:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584541388.928682:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541382/real 1584541382] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541469 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584541388.928693:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584541388.928694:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584541388.928696:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584541388.928704:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541388.928706:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541388.928707:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541388.928708:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541388.928709:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541388.928711:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584541388.928712:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584541388.928714:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23188 00000100:00080000:12.0:1584541388.928717:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584541388.928732:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541388.928733:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541388.928734:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541388.928736:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541388.928737:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541388.928739:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541388.928739:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541388.928740:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541388.959154:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541388.959161:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541388.959163:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541388.959164:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541388.959166:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541388.959169:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541388.959171:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541388.959172:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541469 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584541388.959203:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078462288 new x1661492078462320 00000800:00000100:0.0:1584541395.140527:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541395.141154:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584541395.153226:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541388/real 1584541388] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541475 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584541395.153235:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584541395.153236:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584541395.153237:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584541395.153245:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541395.153246:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541395.153247:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541395.153248:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541395.153249:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541395.153251:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584541395.153251:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584541395.153253:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23194 00000100:00080000:12.0:1584541395.153255:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584541395.153270:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541395.153271:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541395.153273:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541395.153274:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541395.153274:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541395.153287:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541395.153287:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541395.153288:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541395.193052:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584541395.193058:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584541395.193060:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584541395.193061:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584541395.193063:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584541395.193066:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584541395.193067:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584541395.193069:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541475 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584541395.193099:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078462320 new x1661492078462352 00000800:00020000:4.0:1584541400.916152:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584541400.916155:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584541400.916157:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541400.916563:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584541400.928624:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541395/real 1584541395] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541482 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584541400.928633:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584541400.928634:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584541400.928635:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584541400.928652:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541400.928654:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541400.928660:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541400.928661:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541400.928662:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541400.928663:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584541400.928664:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584541400.928666:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23201 00000100:00080000:12.0:1584541400.928668:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584541400.928676:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541400.928677:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541400.928678:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541400.928679:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541400.928680:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541400.928682:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541400.928682:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541400.928683:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541400.977929:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584541400.977935:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584541400.977937:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584541400.977938:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584541400.977939:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584541400.977942:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584541400.977943:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584541400.977945:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541482 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584541400.977974:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078462352 new x1661492078462432 00000800:00020000:2.0:1584541406.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541406.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541406.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541406.916543:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584541406.928595:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541400/real 1584541400] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541487 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584541406.928603:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584541406.928604:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584541406.928605:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584541406.928610:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541406.928612:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541406.928613:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541406.928614:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541406.928614:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541406.928616:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584541406.928616:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584541406.928619:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23206 00000100:00080000:12.0:1584541406.928621:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584541406.928629:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541406.928631:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541406.928631:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541406.928632:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541406.928633:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541406.928634:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541406.928635:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541406.928636:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541406.979627:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584541406.979632:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584541406.979633:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584541406.979634:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584541406.979635:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584541406.979637:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584541406.979639:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584541406.979641:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541487 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584541406.979674:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078462432 new x1661492078462464 00000800:00020000:2.0:1584541412.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541412.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541412.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541412.916492:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584541412.928544:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541406/real 1584541406] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541493 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584541412.928551:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584541412.928552:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584541412.928554:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584541412.928560:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541412.928561:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541412.928562:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541412.928563:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541412.928564:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541412.928567:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584541412.928567:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584541412.928569:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23212 00000100:00080000:12.0:1584541412.928571:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584541412.928580:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541412.928582:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541412.928582:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541412.928584:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541412.928584:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541412.928585:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541412.928586:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541412.928587:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541412.958485:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584541412.958489:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584541412.958491:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584541412.958492:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584541412.958493:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584541412.958495:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584541412.958496:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584541412.958498:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541493 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584541412.958507:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078462464 new x1661492078462512 00000800:00000100:0.0:1584541419.141081:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541419.141520:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584541419.153568:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541412/real 1584541412] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541499 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584541419.197364:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584541419.197365:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584541419.197366:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584541419.197381:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541419.197382:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541419.197383:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541419.197385:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541419.197385:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541419.197387:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584541419.197387:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584541419.197389:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23218 00000100:00080000:12.0:1584541419.197391:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584541419.197399:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541419.197400:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541419.197401:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541419.197402:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541419.197402:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541419.197403:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541419.197404:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541419.197405:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541419.212220:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584541419.212224:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584541419.212226:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584541419.212226:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584541419.212228:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584541419.212229:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584541419.212231:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584541419.212233:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541499 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584541419.212262:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078462512 new x1661492078462544 00000800:00020000:2.0:1584541424.916050:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541424.916052:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541424.916054:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541424.916588:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584541424.928641:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541419/real 1584541419] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541506 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584541424.928649:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584541424.928650:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584541424.928651:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584541424.928664:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541424.928666:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541424.928671:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541424.928673:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541424.928673:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541424.928675:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584541424.928675:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584541424.928677:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23225 00000100:00080000:12.0:1584541424.928679:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584541424.928684:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541424.928685:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541424.928686:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541424.928687:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541424.928688:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541424.928689:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541424.928689:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541424.928691:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541424.974074:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584541424.974078:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584541424.974079:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584541424.974080:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584541424.974081:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584541424.974083:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584541424.974085:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584541424.974086:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541506 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584541424.974096:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078462544 new x1661492078462608 00000800:00020000:2.0:1584541430.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541430.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541430.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541430.916488:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584541430.928535:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541424/real 1584541424] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541511 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584541430.928542:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584541430.928543:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584541430.928545:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584541430.928551:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541430.928552:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541430.928553:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541430.928554:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541430.928555:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541430.928556:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584541430.928557:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584541430.928558:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23230 00000100:00080000:12.0:1584541430.928560:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584541430.928569:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541430.928570:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541430.928571:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541430.928572:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541430.928572:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541430.928574:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541430.928574:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541430.928575:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541430.973936:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584541430.973941:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584541430.973942:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584541430.973943:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584541430.973945:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584541430.973947:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584541430.973948:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584541430.973950:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541511 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584541430.973958:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078462608 new x1661492078462656 00000800:00020000:2.0:1584541436.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541436.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541436.916083:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541436.916485:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584541436.928535:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541430/real 1584541430] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541517 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584541436.928547:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584541436.928548:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584541436.928550:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584541436.928554:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541436.928556:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541436.928557:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541436.928558:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541436.928558:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541436.928560:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584541436.928560:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584541436.928562:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23236 00000100:00080000:12.0:1584541436.928564:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584541436.928572:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541436.928573:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541436.928574:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541436.928575:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541436.928576:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541436.928577:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541436.928577:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541436.928579:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541436.968086:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584541436.968090:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584541436.968092:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584541436.968093:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584541436.968094:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584541436.968096:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584541436.968097:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584541436.968099:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541517 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584541436.968108:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078462656 new x1661492078462688 00000800:00000100:0.0:1584541443.141594:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541443.142042:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584541443.154095:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541436/real 1584541436] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541523 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584541443.154104:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584541443.179449:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584541443.179451:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584541443.179468:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541443.179470:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541443.179470:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541443.179471:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541443.179472:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541443.179474:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584541443.179475:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584541443.179476:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23242 00000100:00080000:12.0:1584541443.179479:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584541443.179488:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541443.179490:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541443.179490:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541443.179491:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541443.179492:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541443.179493:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541443.179495:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541443.179507:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541443.216992:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584541443.216998:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584541443.217000:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584541443.217001:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584541443.217003:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584541443.217005:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584541443.217007:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584541443.234793:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541523 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584541443.234817:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078462688 new x1661492078462736 00000800:00020000:2.0:1584541448.916056:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541448.916059:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541448.916061:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541448.916444:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584541448.928498:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541443/real 1584541443] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541530 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584541448.928505:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584541448.928506:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584541448.928508:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584541448.928522:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541448.928524:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541448.928530:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541448.928531:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541448.928532:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541448.928534:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584541448.928535:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584541448.928536:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23249 00000100:00080000:14.0:1584541448.928539:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584541448.928544:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541448.928546:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541448.928546:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541448.928547:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541448.928548:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541448.928549:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541448.928550:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541448.928551:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541448.980716:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584541448.980720:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584541448.980722:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584541448.980723:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584541448.980724:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584541448.980726:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584541448.980728:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584541448.980729:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541530 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584541448.980738:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078462736 new x1661492078462800 00000800:00020000:2.0:1584541454.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541454.916082:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541454.916084:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541454.916536:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584541454.928592:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541448/real 1584541448] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541535 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584541454.928600:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584541454.928601:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584541454.928603:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584541454.928608:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541454.928610:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541454.928611:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541454.928613:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541454.928614:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541454.928616:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584541454.928616:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584541454.928618:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23254 00000100:00080000:14.0:1584541454.928620:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584541454.928631:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541454.928632:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541454.928633:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541454.928634:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541454.928635:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541454.928636:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541454.928637:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541454.928638:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541454.968946:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584541454.968950:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584541454.968952:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584541454.968953:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584541454.968954:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584541454.968957:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584541454.968958:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584541454.968960:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541535 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584541454.968969:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078462800 new x1661492078462832 00000800:00020000:4.0:1584541460.916095:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584541460.916098:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584541460.916101:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541460.916612:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584541460.928672:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541454/real 1584541454] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541541 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584541460.928683:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584541460.928685:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584541460.928686:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584541460.928693:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541460.928696:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541460.928697:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541460.928698:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541460.928699:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541460.928700:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584541460.928701:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584541460.928704:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23260 00000100:00080000:14.0:1584541460.928707:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584541460.928721:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541460.928723:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541460.928723:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541460.928725:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541460.928725:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541460.928727:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541460.928727:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541460.928729:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541460.988839:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584541460.988846:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584541460.988849:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584541460.988850:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584541460.988852:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584541460.988854:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584541460.988856:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584541460.988858:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541541 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584541460.988867:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078462832 new x1661492078462880 00000800:00000100:0.0:1584541467.140237:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541467.140647:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584541467.152700:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541460/real 1584541460] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541547 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584541467.152710:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584541467.152711:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584541467.152714:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584541467.152721:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541467.152723:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541467.152723:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541467.152725:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541467.152725:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541467.152727:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584541467.152727:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584541467.152729:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23266 00000100:00080000:14.0:1584541467.152732:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584541467.152745:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541467.152747:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541467.152747:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541467.152749:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541467.152749:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541467.152750:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541467.152751:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541467.152752:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541467.209117:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584541467.209124:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584541467.209126:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584541467.209127:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584541467.209128:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584541467.209131:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584541467.209133:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584541467.209134:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541547 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584541467.209143:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078462880 new x1661492078462912 00000800:00020000:6.0:1584541472.916071:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584541472.916074:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584541472.916076:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541472.916516:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:14.0:1584541472.928574:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541467/real 1584541467] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541554 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:14.0:1584541472.928588:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:14.0:1584541472.928590:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:14.0:1584541472.928592:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:14.0:1584541472.928608:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541472.928610:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541472.928615:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541472.928616:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541472.928617:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541472.928619:0:4057:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:14.0:1584541472.928619:0:4057:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:14.0:1584541472.928621:0:4057:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23273 00000100:00080000:14.0:1584541472.928624:0:4057:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:14.0:1584541472.928632:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541472.928633:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541472.928634:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541472.928635:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541472.928636:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541472.928637:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541472.928638:0:4057:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:14.0:1584541472.928639:0:4057:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541472.977346:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584541472.977353:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584541472.977355:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584541472.977356:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584541472.977358:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584541472.977361:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584541472.977362:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584541472.977364:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541554 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584541472.977374:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078462912 new x1661492078462992 00000800:00020000:4.0:1584541478.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584541478.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584541478.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541478.916493:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584541478.928546:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541472/real 1584541472] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541559 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584541478.928558:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584541478.928559:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584541478.928561:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584541478.928567:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541478.928569:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541478.928569:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541478.928570:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541478.928571:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541478.928572:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584541478.928573:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584541478.928575:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23278 00000100:00080000:16.0:1584541478.928577:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584541478.928587:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541478.928588:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541478.928588:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541478.928590:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541478.928590:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541478.928591:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541478.928592:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541478.928593:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541478.970827:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584541478.970831:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584541478.970832:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584541478.970833:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584541478.970835:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584541478.970837:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584541478.970838:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584541478.970840:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541559 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584541478.970850:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078462992 new x1661492078463024 00000800:00020000:2.0:1584541484.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541484.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541484.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541484.916497:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584541484.928551:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541478/real 1584541478] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541565 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584541484.928559:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584541484.928560:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584541484.928561:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584541484.928567:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541484.928569:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541484.928570:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541484.928571:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541484.928571:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541484.928573:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584541484.928573:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584541484.928575:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23284 00000100:00080000:16.0:1584541484.928577:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584541484.928585:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541484.928586:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541484.928587:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541484.928588:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541484.928589:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541484.928590:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541484.928591:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541484.928593:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541484.974242:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584541484.974246:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584541484.974248:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584541484.974248:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584541484.974250:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584541484.974252:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584541484.974253:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584541484.974255:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541565 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584541484.974284:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078463024 new x1661492078463056 00000800:00000100:0.0:1584541491.140595:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541491.141034:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584541491.153088:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541484/real 1584541484] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541571 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584541491.153096:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584541491.153097:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584541491.153098:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584541491.153105:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541491.153106:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541491.153107:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541491.153108:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541491.153109:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541491.153110:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584541491.153111:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584541491.153112:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23290 00000100:00080000:16.0:1584541491.153114:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584541491.153123:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541491.153124:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541491.153125:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541491.153126:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541491.153128:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541491.153129:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541491.153129:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541491.153131:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541491.204875:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584541491.204879:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584541491.204880:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584541491.204881:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584541491.204883:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584541491.204885:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584541491.204886:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584541491.204888:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541571 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584541491.204897:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078463056 new x1661492078463104 00000800:00020000:2.0:1584541496.916057:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541496.916059:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541496.916061:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541496.916564:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584541496.928622:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541491/real 1584541491] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541578 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584541496.928630:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584541496.928631:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584541496.928632:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584541496.928645:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541496.928647:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541496.928652:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541496.928653:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541496.928654:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541496.928656:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584541496.928656:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584541496.928658:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23297 00000100:00080000:16.0:1584541496.928660:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584541496.928665:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541496.928667:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541496.928668:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541496.928669:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541496.928670:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541496.928671:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541496.928672:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584541496.928673:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584541496.985910:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584541496.985914:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584541496.985916:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584541496.985917:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584541496.985918:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584541496.985920:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584541496.985921:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584541496.985923:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541578 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541496.985952:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078463104 new x1661492078463168 00000800:00020000:6.0:1584541502.916094:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584541502.916097:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584541502.916099:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541502.916578:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541502.928669:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541496/real 1584541496] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541583 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541502.928690:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541502.928691:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541502.928693:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541502.928699:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541502.928701:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541502.928701:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541502.928703:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541502.928703:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541502.928706:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541502.928707:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541502.928708:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23302 00000100:00080000:18.0:1584541502.928711:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541502.928721:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541502.928722:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541502.928723:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541502.928724:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541502.928725:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541502.928726:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541502.928726:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541502.928728:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584541502.968536:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584541502.968541:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584541502.968543:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584541502.968544:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584541502.968545:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584541502.968548:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584541502.968549:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584541502.968551:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541583 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541502.968562:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078463168 new x1661492078463216 00000800:00020000:2.0:1584541508.916068:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541508.916071:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541508.916074:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541508.916492:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541508.928570:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541502/real 1584541502] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541589 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541508.928579:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541508.928580:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541508.928581:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541508.928588:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541508.928590:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541508.928591:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541508.928593:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541508.928594:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541508.928596:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541508.928596:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541508.928599:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23308 00000100:00080000:18.0:1584541508.928601:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541508.928612:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541508.928614:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541508.928614:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541508.928615:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541508.928616:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541508.928617:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541508.928618:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541508.928619:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584541508.947945:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584541508.947950:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584541508.947953:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584541508.947954:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584541508.947955:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584541508.947957:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584541508.947959:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584541508.947961:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541589 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541508.947991:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078463216 new x1661492078463248 00000800:00000100:0.0:1584541515.140537:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541515.141052:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541515.153128:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541508/real 1584541508] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541595 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541515.153140:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541515.153141:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541515.153143:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541515.153152:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541515.153155:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541515.153156:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541515.153157:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541515.153158:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541515.153160:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541515.153161:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541515.153163:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23314 00000100:00080000:18.0:1584541515.153167:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541515.153183:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541515.153184:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541515.153185:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541515.153186:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541515.153187:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541515.153188:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541515.153189:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541515.153190:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584541515.209554:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584541515.209561:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584541515.209563:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584541515.209565:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584541515.209567:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584541515.209569:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584541515.209571:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584541515.209573:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541595 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541515.209604:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078463248 new x1661492078463280 00000800:00020000:6.0:1584541520.916045:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584541520.916048:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584541520.916050:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541520.916533:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541520.928599:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541515/real 1584541515] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541602 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541520.928608:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541520.928609:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541520.928611:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541520.928625:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541520.928627:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541520.928633:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541520.928634:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541520.928635:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541520.928636:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541520.928637:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541520.928639:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23321 00000100:00080000:18.0:1584541520.928641:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541520.928648:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541520.928649:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541520.928650:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541520.928651:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541520.928652:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541520.928653:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541520.928654:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541520.928655:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584541520.954210:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584541520.954229:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584541520.954231:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584541520.954232:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584541520.954233:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584541520.954235:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584541520.954237:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584541520.954239:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541602 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541520.954268:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078463280 new x1661492078463360 00000800:00020000:2.0:1584541526.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541526.916076:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541526.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541526.916555:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541526.928638:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541520/real 1584541520] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541607 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541526.928662:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541526.928663:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541526.928665:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541526.928671:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541526.928673:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541526.928673:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541526.928675:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541526.928675:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541526.928677:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541526.928677:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541526.928679:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23326 00000100:00080000:18.0:1584541526.928681:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541526.928692:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541526.928693:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541526.928694:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541526.928695:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541526.928696:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541526.928697:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541526.928698:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541526.928699:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584541526.970288:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584541526.970298:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584541526.970301:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584541526.970303:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584541526.970306:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584541526.970310:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584541526.970314:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584541526.970318:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541607 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541526.970366:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078463360 new x1661492078463392 00000800:00020000:2.0:1584541532.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541532.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541532.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541532.916527:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541532.928581:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541526/real 1584541526] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541613 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541532.928590:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541532.928591:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541532.928593:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541532.928599:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541532.928601:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541532.928602:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541532.928603:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541532.928604:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541532.928605:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541532.928606:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541532.928608:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23332 00000100:00080000:18.0:1584541532.928610:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541532.928622:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541532.928623:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541532.928624:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541532.928625:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541532.928626:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541532.928627:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541532.928629:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541532.928630:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584541532.962626:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584541532.962631:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584541532.962633:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584541532.962634:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584541532.962636:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584541532.962638:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584541532.962639:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584541532.962641:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541613 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541532.962671:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078463392 new x1661492078463440 00000800:00000100:0.0:1584541539.140895:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541539.141342:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541539.153399:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541532/real 1584541532] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541619 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541539.153410:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541539.153411:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541539.153413:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541539.153420:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541539.153422:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541539.153423:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541539.153424:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541539.153425:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541539.153427:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541539.153427:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541539.153430:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23338 00000100:00080000:18.0:1584541539.153433:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541539.153447:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541539.153448:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541539.153449:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541539.153451:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541539.153452:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541539.153454:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541539.153454:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541539.153456:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584541539.209993:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584541539.210000:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584541539.210002:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584541539.210003:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584541539.210005:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584541539.210008:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584541539.210010:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584541539.210012:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541619 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541539.210046:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078463440 new x1661492078463472 00000800:00020000:2.0:1584541544.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541544.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541544.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541544.916611:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541544.928663:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541539/real 1584541539] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541626 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541544.928671:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541544.928672:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541544.928673:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541544.928688:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541544.928689:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541544.928695:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541544.928696:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541544.928697:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541544.928698:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541544.928699:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541544.928701:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23345 00000100:00080000:18.0:1584541544.928703:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541544.928708:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541544.928710:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541544.928711:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541544.928712:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541544.928713:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541544.928714:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541544.928715:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541544.928716:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584541544.944457:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584541544.944461:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584541544.944462:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584541544.944463:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584541544.944464:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584541544.944466:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584541544.944468:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584541544.944469:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541626 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541544.944498:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078463472 new x1661492078463536 00000800:00020000:2.0:1584541550.916101:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541550.916107:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541550.916111:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541550.916549:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541550.928634:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541544/real 1584541544] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541631 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541550.928653:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541550.928654:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541550.928655:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541550.928662:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541550.928663:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541550.928664:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541550.928665:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541550.928666:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541550.928667:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541550.928668:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541550.928670:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23350 00000100:00080000:18.0:1584541550.928672:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541550.928683:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541550.928684:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541550.928685:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541550.928686:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541550.928687:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541550.928688:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541550.928688:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541550.928689:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584541550.934573:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584541550.934577:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584541550.934578:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584541550.934579:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584541550.934581:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584541550.934583:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584541550.934584:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584541550.934586:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541631 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541550.934620:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078463536 new x1661492078463584 00000800:00020000:2.0:1584541556.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541556.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541556.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541556.916551:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541556.928603:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541550/real 1584541550] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541637 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541556.928611:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541556.928612:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541556.928613:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541556.928619:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541556.928621:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541556.928621:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541556.928623:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541556.928623:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541556.928625:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541556.928625:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541556.928628:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23356 00000100:00080000:18.0:1584541556.928630:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541556.928638:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541556.928639:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541556.928640:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541556.928641:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541556.928642:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541556.928643:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541556.928644:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541556.928645:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541556.982061:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584541556.982065:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584541556.982067:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584541556.982068:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584541556.982069:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584541556.982071:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584541556.982073:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584541556.982074:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541637 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541556.982111:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078463584 new x1661492078463632 00000800:00000100:0.0:1584541563.141619:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541563.142072:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541563.154145:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541556/real 1584541556] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541643 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541563.154155:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541563.154156:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541563.154158:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541563.154165:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541563.154167:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541563.154167:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541563.154168:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541563.154169:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541563.154172:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541563.154173:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541563.154175:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23362 00000100:00080000:18.0:1584541563.154178:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541563.154189:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541563.154190:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541563.154191:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541563.154193:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541563.154193:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541563.154195:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541563.154195:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541563.154197:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541563.204021:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584541563.204040:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584541563.204042:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584541563.204043:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584541563.204045:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584541563.204047:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584541563.204049:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584541563.204050:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541643 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541563.204080:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078463632 new x1661492078463680 00000400:00000080:2.0:1584541567.661154:0:181247:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00020000:2.0:1584541568.916067:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541568.916070:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541568.916072:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541568.916492:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541568.928560:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541563/real 1584541563] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541650 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541568.928570:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541568.928571:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541568.928573:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541568.928593:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541568.928595:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541568.928601:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541568.928603:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541568.928604:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541568.928605:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541568.928606:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541568.928608:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23369 00000100:00080000:18.0:1584541568.928611:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541568.928618:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541568.928619:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541568.928620:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541568.928622:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541568.928622:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541568.928624:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541568.928624:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541568.928626:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541568.975096:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584541568.975103:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584541568.975105:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584541568.975106:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584541568.975107:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584541568.975110:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584541568.975111:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584541568.975113:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541650 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541568.975143:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078463680 new x1661492078463744 00000400:00000080:3.0:1584541571.323743:0:181295:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00020000:4.0:1584541574.916092:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584541574.916095:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (3): c: 7, oc: 0, rc: 8 00000800:00000100:4.0:1584541574.916097:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541574.916554:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541574.928648:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541568/real 1584541568] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541655 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541574.928668:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541574.928670:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541574.928671:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541574.928679:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541574.928683:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541574.928683:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541574.928684:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541574.928685:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541574.928687:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541574.928688:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541574.928689:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23374 00000100:00080000:18.0:1584541574.928692:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541574.928705:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541574.928707:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541574.928707:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541574.928709:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541574.928709:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541574.928711:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541574.928711:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541574.928712:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541574.984496:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584541574.984506:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584541574.984510:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584541574.984512:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584541574.984515:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584541574.984521:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584541574.984525:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584541574.984529:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541655 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541574.984582:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078463744 new x1661492078463776 00000400:00000080:2.0:1584541575.779116:0:181306:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00020000:8.0:1584541580.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584541580.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584541580.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541580.916505:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541580.928590:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541574/real 1584541574] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541661 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541580.928601:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541580.928602:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541580.928605:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541580.928614:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541580.928616:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541580.928617:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541580.928618:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541580.928618:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541580.928620:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541580.928621:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541580.928623:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23380 00000100:00080000:18.0:1584541580.928626:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541580.928639:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541580.928640:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541580.928641:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541580.928642:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541580.928643:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541580.928644:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541580.928645:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541580.928646:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541580.934677:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584541580.934685:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584541580.934687:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584541580.934688:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584541580.934690:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584541580.934692:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584541580.934694:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584541580.934695:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541661 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541580.934724:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078463776 new x1661492078463824 00000800:00000100:0.0:1584541587.140130:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541587.140569:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541587.152640:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541580/real 1584541580] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541667 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541587.152653:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541587.152655:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541587.152656:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541587.152663:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541587.152665:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541587.152665:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541587.152666:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541587.152667:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541587.152669:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541587.152670:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541587.152672:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23386 00000100:00080000:18.0:1584541587.152674:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541587.152687:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541587.152688:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541587.152689:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541587.152690:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541587.152690:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541587.152692:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541587.152704:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541587.152705:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541587.158852:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584541587.158858:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584541587.158860:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584541587.158860:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584541587.158862:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584541587.158864:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584541587.158865:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584541587.158867:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541667 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541587.158897:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078463824 new x1661492078463856 00000800:00020000:2.0:1584541592.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541592.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541592.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541592.916473:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541592.928554:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541587/real 1584541587] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541674 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541592.928564:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541592.928565:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541592.928567:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541592.928585:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541592.928588:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541592.928594:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541592.928595:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541592.928596:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541592.928598:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541592.928599:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541592.928601:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23393 00000100:00080000:18.0:1584541592.928604:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541592.928611:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541592.928612:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541592.928613:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541592.928615:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541592.928615:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541592.928617:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541592.928619:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541592.928620:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541592.972568:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584541592.972574:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584541592.972576:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584541592.972577:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584541592.972579:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584541592.972582:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584541592.972583:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584541592.972585:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541674 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541592.972615:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078463856 new x1661492078463936 00000800:00020000:2.0:1584541598.916059:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541598.916062:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541598.916064:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541598.916443:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541598.928531:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541592/real 1584541592] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541679 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541598.928549:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541598.928551:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541598.928552:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541598.928558:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541598.928559:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541598.928560:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541598.928561:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541598.928562:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541598.928563:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541598.928564:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541598.928565:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23398 00000100:00080000:18.0:1584541598.928567:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541598.928576:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541598.928578:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541598.928578:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541598.928579:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541598.928581:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541598.928582:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541598.928583:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541598.928584:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541598.982077:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584541598.982081:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584541598.982082:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584541598.982083:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584541598.982084:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584541598.982086:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584541598.982087:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584541598.982089:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541679 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541598.982118:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078463936 new x1661492078463968 00000800:00020000:2.0:1584541604.916052:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541604.916054:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541604.916056:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541604.916469:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541604.928520:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541598/real 1584541598] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541685 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541604.928527:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541604.928529:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541604.928530:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541604.928535:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541604.928536:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541604.928537:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541604.928538:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541604.928539:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541604.928540:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541604.928541:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541604.928542:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23404 00000100:00080000:18.0:1584541604.928544:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541604.928552:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541604.928554:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541604.928555:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541604.928556:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541604.928557:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541604.928558:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541604.928559:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541604.928560:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541604.981526:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584541604.981530:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584541604.981532:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584541604.981533:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584541604.981534:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584541604.981536:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584541604.981537:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584541604.981539:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541685 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541604.981568:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078463968 new x1661492078464000 00000800:00000100:0.0:1584541611.140664:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541611.141208:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541611.153284:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541604/real 1584541604] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541691 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541611.153291:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541611.153291:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541611.153293:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541611.153298:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541611.153300:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541611.153301:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541611.153302:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541611.153302:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541611.153304:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541611.153304:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541611.153306:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23410 00000100:00080000:18.0:1584541611.153308:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541611.153316:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541611.153318:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541611.153318:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541611.153319:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541611.153320:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541611.153321:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541611.153321:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541611.153334:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541611.207303:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:4.0:1584541611.207307:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:4.0:1584541611.207309:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:4.0:1584541611.207309:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:4.0:1584541611.207311:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:4.0:1584541611.207313:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:4.0:1584541611.207315:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:4.0:1584541611.207317:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541691 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584541611.207352:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078464000 new x1661492078464048 00000800:00020000:4.0:1584541616.916116:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584541616.916119:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584541616.916121:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541616.916534:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584541616.928584:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541611/real 1584541611] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541698 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584541616.928592:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584541616.928593:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584541616.928594:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584541616.928606:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541616.928608:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541616.928613:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541616.928614:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541616.928615:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541616.928616:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584541616.928617:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584541616.928619:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23417 00000100:00080000:18.0:1584541616.928621:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584541616.928626:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541616.928628:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541616.928628:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541616.928630:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541616.928630:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541616.928631:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584541616.928632:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584541616.928633:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541616.969477:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584541616.969480:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584541616.969482:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584541616.969482:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584541616.969484:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584541616.969485:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584541616.969487:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584541616.969489:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541698 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584541616.969497:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078464048 new x1661492078464112 00000800:00020000:8.0:1584541622.916079:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584541622.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584541622.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541622.916519:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584541622.928625:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541616/real 1584541616] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541703 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584541622.928635:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584541622.928636:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584541622.928638:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584541622.928646:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541622.928648:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541622.928649:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541622.928650:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541622.928651:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541622.928652:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584541622.928653:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584541622.928655:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23422 00000100:00080000:20.0:1584541622.928660:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584541622.928676:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541622.928677:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541622.928678:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541622.928679:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541622.928680:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541622.928681:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541622.928681:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541622.928694:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541622.944443:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584541622.944449:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584541622.944451:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584541622.944452:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584541622.944455:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584541622.944458:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584541622.944460:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584541622.944462:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541703 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584541622.944491:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078464112 new x1661492078464160 00000800:00020000:2.0:1584541628.916066:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541628.916069:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541628.916071:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541628.916622:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584541628.928740:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541622/real 1584541622] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541709 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584541628.928750:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584541628.928751:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584541628.928753:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584541628.928761:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541628.928763:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541628.928764:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541628.928765:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541628.928766:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541628.928767:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584541628.928769:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584541628.928771:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23428 00000100:00080000:20.0:1584541628.928774:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584541628.928788:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541628.928789:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541628.928790:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541628.928791:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541628.928792:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541628.928793:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541628.928794:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541628.928795:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541628.934912:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584541628.934918:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584541628.934920:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584541628.934921:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584541628.934923:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584541628.934926:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584541628.934927:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584541628.934929:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541709 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584541628.934959:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078464160 new x1661492078464192 00000800:00000100:0.0:1584541635.140804:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541635.141305:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584541635.153381:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541628/real 1584541628] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541715 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584541635.153389:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584541635.153391:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584541635.153392:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584541635.153399:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541635.153401:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541635.153402:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541635.153403:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541635.153404:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541635.153406:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584541635.153407:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584541635.153408:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23434 00000100:00080000:20.0:1584541635.153411:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584541635.153425:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541635.153426:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541635.153427:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541635.153428:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541635.153428:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541635.153430:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541635.153430:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541635.153442:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541635.197664:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:10.0:1584541635.197670:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:10.0:1584541635.197672:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:10.0:1584541635.197673:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:10.0:1584541635.197675:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:10.0:1584541635.197677:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:10.0:1584541635.197678:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584541635.197680:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541715 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:20.0:1584541635.197714:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078464192 new x1661492078464224 00000800:00020000:8.0:1584541640.916081:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584541640.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584541640.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541640.916590:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:20.0:1584541640.928643:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541635/real 1584541635] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541722 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:20.0:1584541640.928652:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:20.0:1584541640.928653:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:20.0:1584541640.928654:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:20.0:1584541640.928669:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541640.928670:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541640.928677:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541640.928678:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541640.928679:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541640.928680:0:1231:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:20.0:1584541640.928681:0:1231:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:20.0:1584541640.928683:0:1231:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23441 00000100:00080000:20.0:1584541640.928685:0:1231:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:20.0:1584541640.928692:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541640.928693:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541640.928694:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541640.928695:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541640.928695:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541640.928697:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584541640.928697:0:1231:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:20.0:1584541640.928698:0:1231:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000800:00000400:4.0:1584541642.916078:0:25711:0:(o2iblnd_cb.c:3381:kiblnd_check_conns()) Timed out tx for 10.10.2.22@o2ib: 0 seconds 00000400:00000100:4.0:1584541642.927251:0:25711:0:(lib-msg.c:705:lnet_health_check()) msg 0@<0:0>->10.10.2.22@o2ib exceeded retry count 0 00000100:00000400:6.0:1584541642.927282:0:25729:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541640/real 1584541642] req@ffff8d339bbec800 x1661492078464272/t0(0) o400->public1-OST0001-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 224/224 e 0 to 1 dl 1584541653 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 00000100:02000400:6.0:1584541642.927291:0:25729:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection to public1-OST0001 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584541642.927292:0:25729:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584541642.927293:0:25729:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0001_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584541642.927301:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541642.927303:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541642.927303:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541642.927305:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:6.0:1584541642.927306:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584541642.927307:0:42:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 22686 00000100:00080000:6.0:1584541642.927309:0:42:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584541642.927321:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541642.927322:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541642.927323:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541642.927324:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541642.927325:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541642.927326:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541642.927327:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541642.927328:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541642.927328:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541642.927329:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000800:00000400:4.0:1584541643.916083:0:25711:0:(o2iblnd_cb.c:3381:kiblnd_check_conns()) Timed out tx for 10.10.2.22@o2ib: 1 seconds 00000400:00000100:4.0:1584541643.927254:0:25711:0:(lib-msg.c:705:lnet_health_check()) msg 0@<0:0>->10.10.2.22@o2ib exceeded retry count 0 00000100:00080000:10.0:1584541643.927286:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:10.0:1584541643.927288:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0001_UUID on 10.10.2.22@o2ib failed (-110) 00000800:00000400:4.0:1584541646.916074:0:25711:0:(o2iblnd_cb.c:3381:kiblnd_check_conns()) Timed out tx for 10.10.2.22@o2ib: 0 seconds 00000400:00000100:4.0:1584541646.927244:0:25711:0:(lib-msg.c:705:lnet_health_check()) msg 0@<0:0>->10.10.2.22@o2ib exceeded retry count 0 00000100:00080000:12.0:1584541646.927249:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:12.0:1584541646.927251:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0003_UUID on 10.10.2.22@o2ib failed (-110) 00000400:00000080:2.0:1584541658.308923:0:181565:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541660.522152:0:181570:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541662.154876:0:181574:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541663.218659:0:181577:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541664.194733:0:181580:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541665.275185:0:181583:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541666.282261:0:181585:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541667.307071:0:181588:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000100:00080000:6.0:1584541667.993147:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541667.993149:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541667.993150:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:6.0:1584541667.993152:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:6.0:1584541667.993153:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584541667.993156:0:42:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23448 00000100:00080000:6.0:1584541667.993157:0:42:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 22662 00000100:00080000:6.0:1584541667.993160:0:42:0:(import.c:596:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0001 (at 10.10.2.21@o2ib) 00000100:00080000:6.0:1584541667.993161:0:42:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.21@o2ib/10.10.2.21@o2ib 00000100:00080000:6.0:1584541667.993168:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541667.993171:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541667.993172:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:6.0:1584541667.993173:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584541667.993174:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584541667.993175:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23446 00000100:00080000:6.0:1584541667.993176:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 23038 00000100:00080000:6.0:1584541667.993178:0:42:0:(import.c:596:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0003 (at 10.10.2.21@o2ib) 00000100:00080000:6.0:1584541667.993179:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.21@o2ib/10.10.2.21@o2ib 00000100:00080000:6.0:1584541667.993183:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541667.993184:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541667.993184:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541667.993186:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541667.993186:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541667.993188:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541667.993188:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541667.993189:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541667.993330:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:12.0:1584541667.993332:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0001_UUID on 10.10.2.21@o2ib failed (-19) 00000100:00080000:12.0:1584541667.993336:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:12.0:1584541667.993337:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0003_UUID on 10.10.2.21@o2ib failed (-19) 00000400:00000080:2.0:1584541668.618624:0:181591:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541669.708347:0:181594:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541670.659636:0:181597:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541671.635093:0:181600:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541672.677011:0:181602:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541673.619764:0:181605:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541674.811005:0:181608:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541675.724170:0:181611:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541676.619264:0:181613:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:12.0:1584541678.370812:0:181617:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541681.260515:0:181637:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541682.539616:0:181640:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541683.411533:0:181643:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541684.760348:0:181646:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541685.843127:0:181649:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541687.031714:0:181652:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541687.956866:0:181655:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000100:00080000:6.0:1584541693.081425:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541693.081428:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541693.081429:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:6.0:1584541693.081431:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:6.0:1584541693.081433:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584541693.081438:0:42:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23448 00000100:00080000:6.0:1584541693.081439:0:42:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 23473 00000100:00080000:6.0:1584541693.081441:0:42:0:(import.c:570:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: tried all connections, increasing latency to 6s 00000100:00080000:6.0:1584541693.081444:0:42:0:(import.c:596:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0001 (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584541693.081445:0:42:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584541693.081454:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541693.081456:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541693.081456:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:6.0:1584541693.081458:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584541693.081458:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584541693.081460:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23446 00000100:00080000:6.0:1584541693.081461:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 23473 00000100:00080000:6.0:1584541693.081462:0:42:0:(import.c:570:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: tried all connections, increasing latency to 6s 00000100:00080000:6.0:1584541693.081463:0:42:0:(import.c:596:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0003 (at 10.10.2.22@o2ib) 00000100:00080000:6.0:1584541693.081465:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584541693.081468:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541693.081470:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541693.081470:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541693.081472:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541693.081472:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541693.081474:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541693.081474:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541693.081476:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541693.081654:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584541693.081662:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584541693.081665:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584541693.081666:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584541693.081668:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584541693.081670:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584541693.081672:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584541693.081674:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541722 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584541693.081683:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584541693.081685:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584541693.081686:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0001_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584541693.081687:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078464224 new x1661492078464464 00000100:00080000:12.0:1584541693.081687:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584541693.081687:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584541693.081689:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584541693.081690:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000800:00020000:18.0:1584541698.916046:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:18.0:1584541698.916049:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:18.0:1584541698.916052:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541698.916445:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584541698.928560:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541693/real 1584541693] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541780 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584541698.928571:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584541698.928572:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584541698.928575:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584541698.928583:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541698.928586:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541698.928587:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541698.928588:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541698.928589:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541698.928590:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584541698.928591:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584541698.928594:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23498 00000100:00080000:22.0:1584541698.928597:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584541698.928614:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541698.928616:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541698.928617:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541698.928618:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541698.928619:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541698.928620:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541698.928621:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541698.928622:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541698.934386:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:12.0:1584541698.934394:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:12.0:1584541698.934397:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:12.0:1584541698.934398:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:12.0:1584541698.934400:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:12.0:1584541698.934402:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:12.0:1584541698.934404:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584541698.934406:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541780 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584541698.934436:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078464464 new x1661492078464512 00000800:00020000:4.0:1584541704.916044:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584541704.916047:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584541704.916049:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541704.916522:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584541704.928576:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541698/real 1584541698] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541785 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584541704.928586:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584541704.928587:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584541704.928590:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584541704.928597:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541704.928599:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541704.928600:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541704.928601:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541704.928602:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541704.928604:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584541704.928605:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584541704.928607:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23504 00000100:00080000:22.0:1584541704.928610:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584541704.928625:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541704.928626:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541704.928627:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541704.928628:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541704.928629:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541704.928630:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541704.928631:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541704.928632:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541704.934545:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584541704.934552:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584541704.934554:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584541704.934556:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584541704.934557:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584541704.934560:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584541704.934561:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584541704.934563:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541785 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584541704.934572:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078464512 new x1661492078464544 00000400:00000080:2.0:1584541706.826490:0:181760:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541707.827507:0:181762:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541708.828312:0:181772:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541709.829379:0:181775:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541710.830480:0:181778:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00020000:2.0:1584541710.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541710.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (0): c: 3, oc: 0, rc: 8 00000800:00000100:2.0:1584541710.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541710.916471:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584541710.928530:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541704/real 1584541704] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541791 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584541710.928544:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584541710.928546:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584541710.928548:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584541710.928555:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541710.928557:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541710.928558:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541710.928559:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541710.928560:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541710.928562:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584541710.928562:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584541710.928565:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23510 00000100:00080000:22.0:1584541710.928568:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584541710.928583:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541710.928584:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541710.928585:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541710.928586:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541710.928587:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541710.928588:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541710.928589:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541710.928590:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541710.934729:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584541710.934735:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584541710.934737:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584541710.934738:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584541710.934740:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584541710.934742:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584541710.934744:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584541710.934746:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541791 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584541710.934775:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078464544 new x1661492078464576 00000400:00000080:2.0:1584541711.831424:0:181781:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541712.832384:0:181784:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541713.833362:0:181786:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541714.834489:0:181789:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541715.835391:0:181792:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541716.836319:0:181795:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00000100:0.0:1584541717.140357:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541717.140768:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584541717.152820:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541710/real 1584541710] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541797 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584541717.152829:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584541717.152830:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584541717.152831:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584541717.152846:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541717.152848:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541717.152854:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541717.152855:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541717.152856:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541717.152858:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584541717.152858:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584541717.152860:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23516 00000100:00080000:22.0:1584541717.152862:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584541717.152869:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541717.152870:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541717.152871:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541717.152872:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541717.152873:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541717.152874:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541717.152876:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541717.152877:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541717.158798:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584541717.158803:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584541717.158805:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584541717.158806:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584541717.158808:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584541717.158810:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584541717.158812:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584541717.158814:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541797 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584541717.158823:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078464576 new x1661492078464656 00000400:00000080:2.0:1584541717.837473:0:181798:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541718.838569:0:181800:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541719.839406:0:181803:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541720.840402:0:181806:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541721.841331:0:181809:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541722.842469:0:181811:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00020000:2.0:1584541722.916056:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541722.916058:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (0): c: 2, oc: 0, rc: 8 00000800:00000100:2.0:1584541722.916060:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541722.916657:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584541722.928737:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541717/real 1584541717] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541804 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584541722.928745:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584541722.928746:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584541722.928748:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584541722.928753:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541722.928755:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541722.928756:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541722.928757:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541722.928758:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541722.928759:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584541722.928760:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584541722.928761:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23523 00000100:00080000:22.0:1584541722.928763:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584541722.928772:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541722.928773:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541722.928774:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541722.928775:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541722.928776:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541722.928778:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541722.928778:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541722.928779:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541722.934682:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584541722.934686:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584541722.934687:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584541722.934688:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584541722.934689:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584541722.934691:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584541722.934693:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584541722.934694:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541804 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584541722.934726:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078464656 new x1661492078464688 00000400:00000080:3.0:1584541723.843469:0:181813:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541724.844432:0:181816:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541725.845381:0:181819:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541726.846616:0:181822:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541727.847340:0:181825:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541728.848286:0:181827:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00020000:2.0:1584541728.916070:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541728.916072:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (0): c: 2, oc: 0, rc: 8 00000800:00000100:2.0:1584541728.916074:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541728.916478:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584541728.928532:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541722/real 1584541722] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541809 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584541728.928540:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584541728.928541:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584541728.928543:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584541728.928547:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541728.928548:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541728.928549:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541728.928550:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541728.928551:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541728.928552:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584541728.928553:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584541728.928554:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23528 00000100:00080000:22.0:1584541728.928556:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584541728.928565:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541728.928567:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541728.928568:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541728.928569:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541728.928570:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541728.928571:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541728.928572:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541728.928573:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541728.934296:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584541728.934301:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584541728.934302:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584541728.934303:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584541728.934304:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584541728.934306:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584541728.934308:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584541728.934309:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541809 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584541728.934317:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078464688 new x1661492078464736 00000400:00000080:2.0:1584541729.849398:0:181830:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541730.850604:0:181833:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541731.851426:0:181836:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541732.852419:0:181839:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541733.853390:0:181841:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541734.854649:0:181845:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00020000:2.0:1584541734.916046:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541734.916048:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (0): c: 2, oc: 0, rc: 8 00000800:00000100:2.0:1584541734.916050:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541734.916449:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584541734.928540:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541728/real 1584541728] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541815 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584541734.928548:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584541734.928549:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584541734.928551:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584541734.928557:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541734.928559:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541734.928559:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541734.928561:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541734.928561:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541734.928563:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584541734.928563:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584541734.928565:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23534 00000100:00080000:22.0:1584541734.928567:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584541734.928577:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541734.928578:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541734.928579:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541734.928580:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541734.928580:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541734.928582:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541734.928582:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541734.928583:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541734.967610:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584541734.967613:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584541734.967615:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584541734.967616:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584541734.967617:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584541734.967619:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584541734.967620:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584541734.967622:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541815 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584541734.967656:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078464736 new x1661492078464768 00000400:00000080:2.0:1584541735.855390:0:181848:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541736.856381:0:181851:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:5.0F:1584541737.857410:0:181854:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541738.858512:0:181855:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541739.859440:0:181872:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541740.860479:0:181875:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00000100:0.0:1584541741.140771:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541741.141321:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584541741.153453:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541734/real 1584541734] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541821 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584541741.153462:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584541741.153463:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584541741.153465:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584541741.153498:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541741.153499:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541741.153505:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541741.153506:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541741.153507:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541741.153508:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584541741.153509:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584541741.153512:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23540 00000100:00080000:22.0:1584541741.153515:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584541741.153522:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541741.153523:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541741.153524:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541741.153525:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541741.153526:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541741.153527:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541741.153528:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541741.153529:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541741.208617:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584541741.208622:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584541741.208625:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584541741.208626:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584541741.208627:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584541741.208630:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584541741.208631:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584541741.208633:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541821 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584541741.208662:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078464768 new x1661492078464832 00000400:00000080:3.0:1584541741.861418:0:181896:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541742.862369:0:181936:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541743.863371:0:181938:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541744.864387:0:181941:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541745.865369:0:181944:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541746.866630:0:181952:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00020000:2.0:1584541746.916090:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541746.916096:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (0): c: 2, oc: 0, rc: 8 00000800:00000100:2.0:1584541746.916101:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541746.916569:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584541746.928663:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541741/real 1584541741] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541828 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584541746.928684:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584541746.928686:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584541746.928687:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584541746.928696:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541746.928698:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541746.928699:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541746.928700:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541746.928701:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541746.928704:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584541746.928705:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584541746.928707:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23547 00000100:00080000:22.0:1584541746.928721:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584541746.928735:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541746.928736:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541746.928737:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541746.928739:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541746.928739:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541746.928741:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541746.928741:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541746.928742:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541746.966768:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584541746.966774:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584541746.966776:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584541746.966778:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584541746.966779:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584541746.966782:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584541746.966783:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584541746.966785:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541828 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584541746.966816:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078464832 new x1661492078464880 00000400:00000080:3.0:1584541747.867392:0:181958:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541748.868314:0:181960:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541749.869360:0:181963:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541750.870593:0:181966:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541751.871342:0:181971:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541752.872449:0:181973:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00020000:2.0:1584541752.916063:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541752.938878:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (0): c: 2, oc: 0, rc: 8 00000800:00000100:2.0:1584541752.963059:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541752.963556:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584541752.975617:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541746/real 1584541746] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541833 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584541752.975628:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584541752.975630:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584541752.975632:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584541752.975641:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541752.975643:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541752.975644:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541752.975646:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541752.975647:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541752.975649:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584541752.975650:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584541752.975652:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23552 00000100:00080000:22.0:1584541752.975655:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584541752.975671:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541752.975673:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541752.975674:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541752.975675:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541752.975675:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541752.975677:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541752.975678:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541752.975679:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541753.023159:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584541753.023166:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584541753.023168:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584541753.023170:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584541753.023172:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584541753.023175:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584541753.023176:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584541753.023178:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541833 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584541753.023188:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078464880 new x1661492078464912 00000400:00000080:2.0:1584541753.873554:0:181975:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541754.874735:0:181978:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541755.875449:0:181985:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541756.876418:0:181988:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541757.877452:0:181991:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541758.878291:0:181993:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00020000:2.0:1584541758.916089:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541758.916095:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (0): c: 2, oc: 0, rc: 8 00000800:00000100:2.0:1584541758.916100:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541758.916531:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584541758.928633:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541753/real 1584541753] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541840 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584541758.928654:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584541758.928656:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584541758.928658:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584541758.928665:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541758.928668:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541758.928669:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541758.928670:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541758.928671:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541758.928673:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584541758.928674:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584541758.928676:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23558 00000100:00080000:22.0:1584541758.928678:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584541758.928693:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541758.928694:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541758.928695:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541758.928696:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541758.928697:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541758.928698:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541758.928699:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541758.928700:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541758.967263:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584541758.967273:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584541758.967276:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584541758.967279:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584541758.967282:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584541758.967287:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584541758.967290:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584541758.967294:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541840 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584541758.967342:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078464912 new x1661492078464960 00000400:00000080:2.0:1584541759.879408:0:181996:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541760.880416:0:182000:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541761.881398:0:182003:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541762.882688:0:182006:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541763.883369:0:182009:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541764.884375:0:182011:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00000100:0.0:1584541765.140962:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541765.141401:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584541765.153451:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541758/real 1584541758] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541845 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584541765.153459:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584541765.153460:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584541765.153463:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584541765.153478:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541765.153480:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541765.153485:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541765.153486:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541765.153487:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541765.153488:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584541765.153489:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584541765.153490:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23564 00000100:00080000:22.0:1584541765.153493:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584541765.153498:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541765.153500:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541765.153500:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541765.153502:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541765.153502:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541765.153503:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541765.153504:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541765.153505:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541765.191294:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584541765.191298:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584541765.191300:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584541765.191300:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584541765.191302:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584541765.191304:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584541765.191305:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584541765.191307:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541845 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584541765.191316:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078464960 new x1661492078465024 00000400:00000080:2.0:1584541765.885334:0:182014:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541766.886476:0:182016:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541767.887463:0:182019:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541768.888330:0:182030:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541769.889519:0:182032:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541770.890885:0:182035:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00020000:2.0:1584541770.916062:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541770.916065:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (0): c: 2, oc: 0, rc: 8 00000800:00000100:2.0:1584541770.916067:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541770.916487:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584541770.928578:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541765/real 1584541765] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541852 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584541770.928602:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584541770.928603:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584541770.928604:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584541770.928610:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541770.928612:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541770.928612:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541770.928614:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541770.928614:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541770.928616:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584541770.928616:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584541770.928618:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23571 00000100:00080000:22.0:1584541770.928620:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584541770.928628:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541770.928630:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541770.928630:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541770.928631:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541770.928632:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541770.928633:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541770.928634:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541770.928635:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541770.968715:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584541770.968723:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584541770.968726:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584541770.968728:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584541770.968731:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584541770.968735:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584541770.968739:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584541770.968742:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541852 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584541770.968796:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078465024 new x1661492078465056 00000400:00000080:2.0:1584541771.891473:0:182038:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541772.892398:0:182041:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541773.893469:0:182044:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541774.894394:0:182046:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541775.895370:0:182049:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541776.896383:0:182052:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00020000:2.0:1584541776.916059:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541776.916061:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (0): c: 2, oc: 0, rc: 8 00000800:00000100:2.0:1584541776.916063:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541776.916531:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584541776.928584:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541770/real 1584541770] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541857 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584541776.928592:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584541776.928593:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584541776.928595:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584541776.928601:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541776.928603:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541776.928604:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541776.928605:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541776.928605:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541776.928607:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584541776.928608:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584541776.928609:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23576 00000100:00080000:22.0:1584541776.928611:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584541776.928621:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541776.928622:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541776.928623:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541776.928624:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541776.928625:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541776.928626:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541776.928627:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541776.928629:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541776.971463:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584541776.971467:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584541776.971469:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584541776.971470:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584541776.971471:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584541776.971473:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584541776.971475:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584541776.971477:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541857 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584541776.971505:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078465056 new x1661492078465104 00000400:00000080:2.0:1584541777.897422:0:182055:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541778.898513:0:182058:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541779.899340:0:182060:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:5.0:1584541780.900399:0:182063:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541781.901360:0:182065:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541782.902507:0:182068:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00020000:4.0:1584541782.916093:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584541782.916098:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (0): c: 2, oc: 0, rc: 8 00000800:00000100:4.0:1584541782.916103:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541782.916543:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584541782.928618:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541776/real 1584541776] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541863 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584541782.928636:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584541782.928637:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584541782.928639:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584541782.928644:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541782.928646:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541782.928647:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541782.928648:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541782.928648:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541782.928650:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584541782.928650:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584541782.928652:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23582 00000100:00080000:22.0:1584541782.928654:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584541782.928662:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541782.928663:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541782.928664:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541782.928665:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541782.928666:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541782.928668:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541782.928668:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541782.928680:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541782.974523:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584541782.974527:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584541782.974528:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584541782.974529:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584541782.974530:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584541782.974532:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584541782.974534:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584541782.974535:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541863 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:22.0:1584541782.974565:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078465104 new x1661492078465136 00000400:00000080:2.0:1584541783.903413:0:182071:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541784.904409:0:182073:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541785.905415:0:182076:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541786.906678:0:182079:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541787.907423:0:182082:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541788.908361:0:182085:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00000100:0.0:1584541789.141409:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541789.141862:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:22.0:1584541789.153917:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541782/real 1584541782] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541869 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:22.0:1584541789.153925:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:22.0:1584541789.153926:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:22.0:1584541789.153928:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:22.0:1584541789.153941:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541789.153943:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541789.153947:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541789.153960:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541789.153960:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541789.153962:0:4063:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:22.0:1584541789.153962:0:4063:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:22.0:1584541789.153964:0:4063:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23588 00000100:00080000:22.0:1584541789.153966:0:4063:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:22.0:1584541789.153972:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541789.153973:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541789.153975:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541789.153976:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541789.153976:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541789.153977:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584541789.153978:0:4063:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:22.0:1584541789.153979:0:4063:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541789.179038:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584541789.179042:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584541789.179043:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584541789.179044:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584541789.179046:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584541789.179048:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584541789.179049:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584541789.179051:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541869 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584541789.179060:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078465136 new x1661492078465216 00000400:00000080:2.0:1584541789.909386:0:182087:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541790.910627:0:182090:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541791.911392:0:182093:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541792.912368:0:182096:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541793.913389:0:182099:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541794.914448:0:182101:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00020000:2.0:1584541794.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541794.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (0): c: 2, oc: 0, rc: 8 00000800:00000100:2.0:1584541794.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541794.916528:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584541794.928610:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541789/real 1584541789] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541876 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584541794.928629:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584541794.928630:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584541794.928631:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584541794.928637:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541794.928638:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541794.928639:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541794.928641:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541794.928642:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541794.928643:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584541794.928643:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584541794.928645:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23595 00000100:00080000:4.0:1584541794.928647:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584541794.928656:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541794.928658:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541794.928658:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541794.928659:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541794.928660:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541794.928661:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541794.928662:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541794.928663:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541794.971588:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584541794.971592:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584541794.971594:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584541794.971594:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584541794.971596:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584541794.971610:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584541794.971611:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584541794.971613:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541876 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584541794.971657:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078465216 new x1661492078465248 00000400:00000080:2.0:1584541795.915344:0:182104:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541796.916358:0:182107:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541797.917371:0:182109:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541798.918298:0:182112:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541799.919330:0:182128:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00020000:6.0:1584541800.916108:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584541800.916111:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (1): c: 3, oc: 0, rc: 8 00000800:00000100:6.0:1584541800.916113:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541800.916527:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584541800.928613:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541794/real 1584541794] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541881 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584541800.928624:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584541800.928626:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584541800.928627:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584541800.928633:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541800.928635:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541800.928636:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541800.928637:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541800.928638:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541800.928639:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584541800.928640:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584541800.928642:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23600 00000100:00080000:6.0:1584541800.928644:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584541800.928653:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541800.928655:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541800.928655:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541800.928656:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541800.928658:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541800.928659:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541800.928660:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541800.928661:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000400:00000080:3.0:1584541800.929914:0:182131:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000100:00080000:14.0:1584541800.967389:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584541800.967393:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584541800.967395:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584541800.967395:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584541800.967397:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584541800.967399:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584541800.967401:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584541800.967402:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541881 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584541800.967411:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078465248 new x1661492078465280 00000400:00000080:3.0:1584541800.968596:0:182132:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541800.969875:0:182133:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541800.971207:0:182134:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541800.972498:0:182135:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541800.973798:0:182136:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00020000:2.0:1584541806.916104:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541806.916110:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541806.916115:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541806.916621:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584541806.928710:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541800/real 1584541800] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541887 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584541806.928730:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584541806.928731:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584541806.928733:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584541806.928740:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541806.928742:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541806.928742:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541806.928744:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541806.928744:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541806.928746:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584541806.928747:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584541806.928749:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23606 00000100:00080000:6.0:1584541806.928752:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584541806.928763:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541806.928765:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541806.928766:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541806.928767:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541806.928768:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541806.928769:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541806.928781:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541806.928782:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:14.0:1584541806.987151:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:14.0:1584541806.987157:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:14.0:1584541806.987159:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:14.0:1584541806.987161:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:14.0:1584541806.987162:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:14.0:1584541806.987165:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:14.0:1584541806.987166:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:14.0:1584541806.987168:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541887 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584541806.987182:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078465280 new x1661492078465328 00000800:00000100:0.0:1584541813.142121:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541813.142694:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584541813.154768:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541806/real 1584541806] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541893 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584541813.154777:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584541813.154779:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584541813.154780:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584541813.154799:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541813.154801:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541813.154817:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541813.154818:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541813.154819:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541813.154820:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584541813.154821:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584541813.154823:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23612 00000100:00080000:6.0:1584541813.154836:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584541813.154856:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541813.154858:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541813.154859:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541813.154860:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541813.154861:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541813.154862:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541813.154863:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541813.154864:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000800:00000400:2.0:1584541813.916043:0:25711:0:(o2iblnd_cb.c:3381:kiblnd_check_conns()) Timed out tx for 10.10.2.22@o2ib: 6 seconds 00000400:00000100:2.0:1584541813.927209:0:25711:0:(lib-msg.c:705:lnet_health_check()) msg 0@<0:0>->10.10.2.22@o2ib exceeded retry count 0 00000100:00000400:12.0:1584541813.927218:0:25729:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541813/real 1584541813] req@ffff8d3397591f80 x1661492078465360/t0(0) o400->public1-OST0001-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:28/4 lens 224/224 e 0 to 1 dl 1584541865 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 00000100:02000400:12.0:1584541813.927226:0:25729:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection to public1-OST0001 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584541813.927227:0:25729:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584541813.927228:0:25729:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0001_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584541813.927237:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541813.927239:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541813.927239:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541813.927241:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:12.0:1584541813.927241:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584541813.927243:0:4059:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23498 00000100:00080000:12.0:1584541813.927246:0:4059:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584541813.927255:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541813.927256:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541813.927257:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541813.927259:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541813.927260:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541813.927261:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541813.927262:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541813.927263:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541813.927263:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541813.927265:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000800:00000400:2.0:1584541814.916054:0:25711:0:(o2iblnd_cb.c:3381:kiblnd_check_conns()) Timed out tx for 10.10.2.22@o2ib: 7 seconds 00000400:00000100:2.0:1584541814.916057:0:25711:0:(lib-msg.c:705:lnet_health_check()) msg 0@<0:0>->10.10.2.22@o2ib exceeded retry count 0 00000100:00080000:16.0:1584541814.916083:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:16.0:1584541814.916085:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0001_UUID on 10.10.2.22@o2ib failed (-110) 00000002:00080000:3.0:1584541816.518214:0:182176:0:(mdc_request.c:903:mdc_close()) @@@ matched open req@ffff8d3bbf9b3f00 x1661492078465408/t87776301104(87776301104) o101->public1-MDT0000-mdc-ffff8d3ae8ab1800@10.10.2.11@o2ib:12/10 lens 576/616 e 0 to 0 dl 1584541825 ref 1 fl Complete:RP/4/ffffffff rc 0/-1 00000800:00000400:2.0:1584541818.916074:0:25711:0:(o2iblnd_cb.c:3381:kiblnd_check_conns()) Timed out tx for 10.10.2.22@o2ib: 0 seconds 00000400:00000100:2.0:1584541818.916078:0:25711:0:(lib-msg.c:705:lnet_health_check()) msg 0@<0:0>->10.10.2.22@o2ib exceeded retry count 0 00000100:00080000:16.0:1584541818.916112:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:16.0:1584541818.916114:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0003_UUID on 10.10.2.22@o2ib failed (-110) 00000100:00080000:12.0:1584541839.001151:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541839.001157:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541839.001159:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:12.0:1584541839.001163:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:12.0:1584541839.001165:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584541839.001171:0:4059:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23619 00000100:00080000:12.0:1584541839.001175:0:4059:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 23473 00000100:00080000:12.0:1584541839.001180:0:4059:0:(import.c:596:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0001 (at 10.10.2.21@o2ib) 00000100:00080000:12.0:1584541839.001183:0:4059:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.21@o2ib/10.10.2.21@o2ib 00000100:00080000:12.0:1584541839.001197:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541839.001200:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541839.001202:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:12.0:1584541839.001204:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584541839.001205:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584541839.001211:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23619 00000100:00080000:12.0:1584541839.001213:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 23473 00000100:00080000:12.0:1584541839.001217:0:4059:0:(import.c:596:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0003 (at 10.10.2.21@o2ib) 00000100:00080000:12.0:1584541839.001219:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.21@o2ib/10.10.2.21@o2ib 00000100:00080000:12.0:1584541839.001226:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541839.001229:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541839.001231:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541839.001233:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541839.001235:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541839.001238:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541839.001239:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541839.001242:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541839.001444:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:16.0:1584541839.001448:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0001_UUID on 10.10.2.21@o2ib failed (-19) 00000100:00080000:16.0:1584541839.001453:0:25724:0:(import.c:1296:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:16.0:1584541839.001455:0:25724:0:(import.c:1343:ptlrpc_connect_interpret()) recovery of public1-OST0003_UUID on 10.10.2.21@o2ib failed (-19) 00000100:00080000:12.0:1584541864.089163:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541864.089168:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541864.089170:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:12.0:1584541864.089174:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0001_UUID: starting recovery 00000100:00080000:12.0:1584541864.089176:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584541864.089183:0:4059:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23619 00000100:00080000:12.0:1584541864.089186:0:4059:0:(import.c:528:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 23644 00000100:00080000:12.0:1584541864.089189:0:4059:0:(import.c:570:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: tried all connections, increasing latency to 6s 00000100:00080000:12.0:1584541864.089194:0:4059:0:(import.c:596:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0001 (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584541864.089199:0:4059:0:(import.c:604:import_select_connection()) public1-OST0001-osc-ffff8d3ae8ab1800: import ffff8d3ac9bf4800 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584541864.089213:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541864.089217:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541864.089219:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 1 00000100:00080000:12.0:1584541864.089222:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584541864.089223:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584541864.089226:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23619 00000100:00080000:12.0:1584541864.089229:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.21@o2ib last attempt 23644 00000100:00080000:12.0:1584541864.089230:0:4059:0:(import.c:570:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: tried all connections, increasing latency to 6s 00000100:00080000:12.0:1584541864.089233:0:4059:0:(import.c:596:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection changing to public1-OST0003 (at 10.10.2.22@o2ib) 00000100:00080000:12.0:1584541864.089236:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584541864.089243:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541864.089246:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541864.089247:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541864.089250:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541864.089252:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541864.089268:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584541864.089268:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584541864.089270:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541864.089487:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541864.089495:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541864.089497:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541864.089498:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541864.089501:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541864.089503:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541864.089505:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541864.089507:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541893 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584541864.089518:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0001-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541864.089520:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541864.089521:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0001_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541864.089522:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541864.089523:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d3ac9bf4800 public1-OST0001_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541864.089524:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541864.089525:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0001-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:10.0:1584541864.089540:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078465328 new x1661492078465616 00000800:00020000:10.0:1584541869.916044:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:10.0:1584541869.916047:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:10.0:1584541869.916049:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541869.916593:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584541869.928693:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541864/real 1584541864] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541951 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584541869.928702:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584541869.928704:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584541869.928706:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584541869.928713:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541869.928715:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541869.928715:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541869.928717:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541869.928717:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541869.928719:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584541869.928720:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584541869.928722:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23669 00000100:00080000:10.0:1584541869.928724:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584541869.928737:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541869.928739:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541869.928739:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541869.928740:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541869.928741:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541869.928743:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541869.928744:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541869.928745:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541869.965114:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541869.965120:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541869.965121:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541869.965122:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541869.965124:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541869.965126:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541869.965128:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541869.965129:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541951 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584541869.965159:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078465616 new x1661492078465648 00000800:00000100:0.0:1584541876.140457:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541876.140900:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584541876.152973:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541869/real 1584541869] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541956 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584541876.152984:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584541876.152985:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584541876.152987:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584541876.152995:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541876.152997:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541876.152998:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541876.152999:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541876.153000:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541876.153002:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584541876.153003:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584541876.153005:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23675 00000100:00080000:10.0:1584541876.153008:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584541876.153024:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541876.153026:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541876.153041:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541876.153044:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541876.153045:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541876.153046:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541876.153047:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541876.153048:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541876.182820:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541876.182831:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541876.182834:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541876.182835:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541876.182838:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541876.182840:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541876.182842:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541876.182845:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541956 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584541876.182880:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078465648 new x1661492078465680 00000800:00020000:4.0:1584541881.916046:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584541881.916049:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584541881.916051:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541881.916535:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584541881.928591:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541876/real 1584541876] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541963 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584541881.928601:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584541881.928603:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584541881.928605:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584541881.928612:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541881.928615:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541881.928616:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541881.928617:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541881.928618:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541881.928620:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584541881.928621:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584541881.928623:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23682 00000100:00080000:10.0:1584541881.928626:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584541881.928637:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541881.928639:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541881.928640:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541881.928642:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541881.928642:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541881.928644:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541881.928644:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541881.928646:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541881.981046:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541881.981051:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541881.981053:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541881.981055:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541881.981057:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541881.981060:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541881.981061:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541881.981063:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541963 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584541881.981072:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078465680 new x1661492078465728 00000800:00020000:2.0:1584541887.916145:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541887.916148:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541887.916150:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541887.916569:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584541887.928623:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541881/real 1584541881] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541968 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584541887.928631:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584541887.928633:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584541887.928634:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584541887.928650:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541887.928652:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541887.928660:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541887.928662:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541887.928662:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541887.928664:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584541887.928665:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584541887.928667:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23687 00000100:00080000:10.0:1584541887.928670:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584541887.928677:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541887.928678:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541887.928679:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541887.928681:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541887.928681:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541887.928683:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541887.928683:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541887.928685:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541887.982182:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541887.982188:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541887.982189:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541887.982190:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541887.982192:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541887.982194:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541887.982195:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541887.982197:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541968 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584541887.982206:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078465728 new x1661492078465792 00000800:00020000:2.0:1584541893.916090:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541893.916093:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541893.916095:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541893.916507:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584541893.928565:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541887/real 1584541887] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541974 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584541893.928573:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584541893.928575:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584541893.928576:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584541893.928583:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541893.928584:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541893.928585:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541893.928586:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541893.928587:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541893.928589:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584541893.928590:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584541893.928592:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23693 00000100:00080000:10.0:1584541893.928594:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584541893.928604:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541893.928605:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541893.928606:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541893.928608:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541893.928608:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541893.928609:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541893.928610:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541893.928611:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541893.955064:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541893.955070:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541893.955071:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541893.955072:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541893.955074:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541893.955076:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541893.955078:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541893.955079:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541974 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584541893.955088:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078465792 new x1661492078465840 00000800:00000100:0.0:1584541900.140796:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541900.141219:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584541900.153270:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541893/real 1584541893] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541980 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584541900.153277:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584541900.153278:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584541900.153280:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584541900.153285:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541900.153287:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541900.153288:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541900.153289:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541900.153290:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541900.153292:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584541900.153292:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584541900.153294:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23699 00000100:00080000:10.0:1584541900.153296:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584541900.153304:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541900.153305:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541900.153306:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541900.153307:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541900.153308:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541900.153309:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541900.153310:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541900.153311:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541900.215226:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541900.215230:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541900.215232:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541900.215233:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541900.215234:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541900.215236:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541900.215238:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541900.215239:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541980 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584541900.215269:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078465840 new x1661492078465872 00000800:00020000:2.0:1584541905.916050:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541905.916053:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541905.916055:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541905.916493:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584541905.928544:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541900/real 1584541900] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541987 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584541905.928551:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584541905.928553:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584541905.928554:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584541905.928559:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541905.928561:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541905.928563:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541905.928564:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541905.928565:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541905.928566:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584541905.928567:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584541905.928568:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23706 00000100:00080000:10.0:1584541905.928570:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584541905.928578:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541905.928580:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541905.928580:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541905.928581:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541905.928582:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541905.928583:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541905.928584:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541905.928585:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541905.955638:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541905.955642:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541905.955643:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541905.955644:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541905.955645:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541905.955647:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541905.955649:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541905.955650:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541987 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584541905.955659:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078465872 new x1661492078465904 00000800:00020000:8.0:1584541911.916060:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584541911.916063:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584541911.916065:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541911.916450:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584541911.928503:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541905/real 1584541905] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541992 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584541911.928511:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584541911.928512:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584541911.928514:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584541911.928529:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541911.928531:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541911.928536:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541911.928538:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541911.928538:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541911.928540:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584541911.928540:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584541911.928542:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23711 00000100:00080000:10.0:1584541911.928544:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584541911.928549:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541911.928550:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541911.928551:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541911.928552:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541911.928553:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541911.928554:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541911.928554:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541911.928556:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541911.979859:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541911.979863:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541911.979864:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541911.979865:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541911.979866:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541911.979868:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541911.979870:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541911.979872:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541992 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584541911.979880:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078465904 new x1661492078465984 00000800:00020000:2.0:1584541917.916139:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541917.916142:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541917.916144:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541917.916553:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584541917.928607:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541911/real 1584541911] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541998 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584541917.928615:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584541917.928617:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584541917.928618:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584541917.928624:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541917.928626:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541917.928627:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541917.928628:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541917.928628:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541917.928630:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584541917.928630:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584541917.928632:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23717 00000100:00080000:10.0:1584541917.928634:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584541917.928642:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541917.928643:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541917.928644:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541917.928645:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541917.928646:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541917.928647:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541917.928647:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541917.928648:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541917.934596:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541917.934600:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541917.934603:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541917.934604:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541917.934605:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541917.934607:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541917.934608:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541917.934610:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584541998 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584541917.934618:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078465984 new x1661492078466016 00000800:00000100:0.0:1584541924.141501:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541924.141920:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584541924.153980:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541917/real 1584541917] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542004 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584541924.153991:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584541924.153992:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584541924.153994:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584541924.154002:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541924.154004:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541924.154005:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541924.154006:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541924.154007:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541924.154009:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584541924.154010:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584541924.154012:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23723 00000100:00080000:10.0:1584541924.154015:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584541924.154037:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541924.154039:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541924.154039:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541924.154041:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541924.154041:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541924.154043:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541924.154043:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541924.154045:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541924.160117:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541924.160124:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541924.160126:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541924.160127:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541924.160129:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541924.160132:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541924.160133:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541924.160135:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542004 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584541924.160144:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078466016 new x1661492078466064 00000800:00020000:8.0:1584541929.916040:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584541929.916044:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584541929.916046:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541929.916553:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584541929.928623:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541924/real 1584541924] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542011 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584541929.928634:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584541929.928635:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584541929.928637:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584541929.928645:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541929.928647:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541929.928648:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541929.928649:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541929.928650:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541929.928652:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584541929.928653:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584541929.928655:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23730 00000100:00080000:10.0:1584541929.928658:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584541929.928673:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541929.928674:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541929.928675:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541929.928676:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541929.928677:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541929.928678:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584541929.928680:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584541929.928682:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541929.934674:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541929.934680:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541929.934682:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541929.934683:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541929.934685:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541929.934688:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541929.934689:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541929.934691:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542011 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584541929.934721:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078466064 new x1661492078466096 00000800:00020000:2.0:1584541935.916040:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541935.916043:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541935.916045:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541935.916432:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:2.0:1584541935.928507:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541929/real 1584541929] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542016 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:2.0:1584541935.928516:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1584541935.928518:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1584541935.928519:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1584541935.928539:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584541935.928541:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584541935.928552:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584541935.928553:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584541935.928554:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584541935.928555:0:176857:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:2.0:1584541935.928556:0:176857:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1584541935.928558:0:176857:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23735 00000100:00080000:2.0:1584541935.928560:0:176857:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:2.0:1584541935.928567:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584541935.928569:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584541935.928570:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584541935.928571:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584541935.928572:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584541935.928573:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584541935.928573:0:176857:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1584541935.928575:0:176857:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541935.934350:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541935.934356:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541935.934359:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541935.934361:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541935.934363:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541935.934365:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541935.934366:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541935.934368:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542016 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:2.0:1584541935.934377:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078466096 new x1661492078466160 00000800:00020000:2.0:1584541941.916126:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541941.916129:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584541941.916131:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541941.916564:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584541941.928617:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541935/real 1584541935] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542022 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584541941.928625:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584541941.928626:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584541941.928628:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584541941.928633:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541941.928635:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541941.928636:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541941.928637:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541941.928638:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541941.928639:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584541941.928640:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584541941.928641:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23741 00000100:00080000:4.0:1584541941.928643:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584541941.928653:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541941.928655:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541941.928655:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541941.928656:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541941.928657:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541941.928658:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541941.928659:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541941.928660:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541941.934489:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541941.934494:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541941.934496:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541941.934498:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541941.934499:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541941.934502:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541941.934503:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541941.934505:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542022 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584541941.934536:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078466160 new x1661492078466208 00000400:00000080:2.0:1584541946.124466:0:182650:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00000100:0.0:1584541948.136123:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541948.136561:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584541948.148629:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541941/real 1584541941] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542028 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584541948.148642:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584541948.148643:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584541948.148644:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584541948.148650:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541948.148652:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541948.148652:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541948.148653:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541948.148654:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541948.148655:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584541948.148656:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584541948.148658:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23747 00000100:00080000:4.0:1584541948.148660:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584541948.148668:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541948.148669:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541948.148670:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541948.148671:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541948.148672:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541948.148673:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541948.148673:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541948.148675:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541948.154689:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541948.154693:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541948.154695:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541948.154696:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541948.154697:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541948.154699:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541948.154701:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541948.154702:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542028 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584541948.154731:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078466208 new x1661492078466240 00000400:00000080:3.0:1584541948.158158:0:182663:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541949.795653:0:182667:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541952.276538:0:182672:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00020000:6.0:1584541953.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584541953.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (1): c: 6, oc: 0, rc: 8 00000800:00000100:6.0:1584541953.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541953.916528:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584541953.928609:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541948/real 1584541948] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542035 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584541953.928617:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584541953.928618:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584541953.928619:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584541953.928625:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541953.928626:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541953.928627:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541953.928628:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541953.928629:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541953.928630:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584541953.928631:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584541953.928632:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23754 00000100:00080000:4.0:1584541953.928634:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584541953.928647:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541953.928648:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541953.928649:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541953.928650:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541953.928651:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541953.928652:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541953.928653:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541953.928654:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541953.934669:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541953.934674:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541953.934676:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541953.934677:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541953.934678:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541953.934680:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541953.934682:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541953.934683:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542035 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:4.0:1584541953.934714:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078466240 new x1661492078466288 00000400:00000080:3.0:1584541959.844798:0:182686:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00020000:4.0:1584541959.916074:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584541959.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (0): c: 7, oc: 0, rc: 8 00000800:00000100:4.0:1584541959.916078:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541959.916491:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:4.0:1584541959.928561:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541953/real 1584541953] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542040 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:4.0:1584541959.928569:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:4.0:1584541959.928570:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:4.0:1584541959.928571:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:4.0:1584541959.928585:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541959.928586:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541959.928589:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541959.928590:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541959.928590:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541959.928592:0:170:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:4.0:1584541959.928592:0:170:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:4.0:1584541959.928594:0:170:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23759 00000100:00080000:4.0:1584541959.928596:0:170:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:4.0:1584541959.928601:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541959.928602:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541959.928604:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541959.928605:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541959.928606:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541959.928607:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:4.0:1584541959.928607:0:170:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:4.0:1584541959.928608:0:170:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541959.934614:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541959.934618:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541959.934620:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541959.934621:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541959.934622:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541959.934624:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541959.934625:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541959.934627:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542040 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584541959.934636:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078466288 new x1661492078466352 00000400:00000080:3.0:1584541963.749695:0:182694:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00020000:2.0:1584541965.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541965.916080:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (2): c: 7, oc: 0, rc: 8 00000800:00000100:2.0:1584541965.916082:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541965.916584:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584541965.928662:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541959/real 1584541959] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542046 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584541965.928670:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584541965.928671:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584541965.928673:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584541965.928678:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541965.928680:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541965.928681:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541965.928682:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541965.928684:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541965.928685:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584541965.928686:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584541965.928688:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23765 00000100:00080000:6.0:1584541965.928690:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584541965.928699:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541965.928700:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541965.928701:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541965.928702:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541965.928702:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541965.928703:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541965.928704:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541965.928705:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541965.934457:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541965.934461:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541965.934462:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541965.934463:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541965.934464:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541965.934466:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541965.934468:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541965.934469:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542046 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584541965.934498:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078466352 new x1661492078466384 00000400:00000080:2.0:1584541966.333257:0:182699:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541968.612390:0:182704:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584541970.710495:0:182709:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00000100:0.0:1584541972.136509:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541972.136939:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584541972.148989:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541965/real 1584541965] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542052 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584541972.148997:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584541972.148998:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584541972.148999:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584541972.149005:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541972.149006:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541972.149008:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541972.149009:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541972.149010:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541972.149011:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584541972.149012:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584541972.149014:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23771 00000100:00080000:6.0:1584541972.149016:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584541972.149025:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541972.149037:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541972.149038:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541972.149039:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541972.149039:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541972.149041:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541972.149041:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541972.149042:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541972.154669:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541972.154674:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541972.154677:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541972.154678:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541972.154679:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541972.154682:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541972.154683:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541972.154685:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542052 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584541972.154715:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078466384 new x1661492078466432 00000400:00000080:3.0:1584541972.861264:0:182713:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541975.844644:0:182721:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541977.908596:0:182725:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00020000:2.0:1584541977.916059:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541977.916062:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (0): c: 5, oc: 0, rc: 8 00000800:00000100:2.0:1584541977.916064:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541977.916470:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584541977.928548:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541972/real 1584541972] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542059 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584541977.928556:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584541977.928557:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584541977.928558:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584541977.928565:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541977.928567:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541977.928567:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541977.928568:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541977.928569:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541977.928570:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584541977.928571:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584541977.928573:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23778 00000100:00080000:6.0:1584541977.928575:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584541977.928583:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541977.928584:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541977.928585:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541977.928586:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541977.928587:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541977.928588:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541977.928588:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541977.928589:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541977.934406:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541977.934410:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541977.934412:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541977.934413:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541977.934414:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541977.934416:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541977.934417:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541977.934419:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542059 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584541977.934428:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078466432 new x1661492078466464 00000400:00000080:3.0:1584541979.988860:0:182744:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584541982.174868:0:182770:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00020000:4.0:1584541983.916063:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584541983.916067:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (1): c: 6, oc: 0, rc: 8 00000800:00000100:4.0:1584541983.916069:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541983.916731:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584541983.928846:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541977/real 1584541977] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542064 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584541983.928857:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584541983.928859:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584541983.928861:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584541983.928881:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541983.928883:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541983.928886:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541983.928888:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541983.928888:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541983.928890:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584541983.928891:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584541983.928893:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23783 00000100:00080000:6.0:1584541983.928897:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584541983.928905:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541983.928906:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541983.928907:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541983.928908:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541983.928909:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541983.928910:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541983.928911:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541983.928912:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541983.964273:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541983.964281:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541983.964283:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541983.964284:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541983.964287:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541983.964289:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541983.964291:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541983.964293:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542064 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584541983.964303:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078466464 new x1661492078466544 00000400:00000080:3.0:1584541984.591649:0:182816:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00020000:2.0:1584541989.916275:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584541989.916278:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 7, oc: 0, rc: 8 00000800:00000100:2.0:1584541989.916280:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584541989.916703:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584541989.928762:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541983/real 1584541983] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542070 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584541989.928771:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584541989.928773:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584541989.928775:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584541989.928782:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541989.928784:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541989.928785:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541989.928786:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541989.928787:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541989.928788:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584541989.928789:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584541989.928791:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23789 00000100:00080000:6.0:1584541989.928794:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584541989.928808:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541989.928809:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541989.928810:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541989.928811:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541989.928812:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541989.928813:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541989.928814:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541989.928815:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541989.971040:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541989.971047:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541989.971049:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541989.971050:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541989.971052:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541989.971055:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541989.971056:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541989.971058:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542070 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584541989.971087:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078466544 new x1661492078466576 00000800:00000100:0.0:1584541996.141161:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584541996.141572:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584541996.153635:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541989/real 1584541989] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542076 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584541996.153643:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584541996.153645:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584541996.153646:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584541996.153652:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541996.153654:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541996.153655:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541996.153656:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541996.153657:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541996.153658:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584541996.153659:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584541996.153661:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23795 00000100:00080000:6.0:1584541996.153664:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584541996.153677:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541996.153678:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541996.153679:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541996.153680:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541996.153681:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541996.153682:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584541996.153684:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584541996.153685:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584541996.207614:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584541996.207620:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584541996.207622:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584541996.207623:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584541996.207625:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584541996.207627:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584541996.207629:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584541996.207631:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542076 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584541996.207640:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078466576 new x1661492078466608 00000800:00020000:0.0:1584542001.916108:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584542001.916111:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584542001.916113:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542001.916554:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584542001.928625:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584541996/real 1584541996] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542083 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584542001.928633:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584542001.928634:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584542001.928636:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584542001.928642:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584542001.928644:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584542001.928644:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584542001.928646:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584542001.928646:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584542001.928648:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584542001.928648:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584542001.928650:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23802 00000100:00080000:6.0:1584542001.928653:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584542001.928663:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584542001.928664:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584542001.928665:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584542001.928666:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584542001.928667:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584542001.928669:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584542001.928669:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584542001.928670:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542001.986861:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584542001.986866:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584542001.986868:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584542001.986869:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584542001.986870:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584542001.986873:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584542001.986874:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584542001.986876:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542083 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584542001.986886:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078466608 new x1661492078466656 00000800:00020000:2.0:1584542007.916115:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542007.916117:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542007.916120:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542007.916537:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:6.0:1584542007.928613:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542001/real 1584542001] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542088 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:6.0:1584542007.928621:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:6.0:1584542007.928622:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:6.0:1584542007.928624:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:6.0:1584542007.928638:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584542007.928640:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584542007.928645:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584542007.928646:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584542007.928647:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584542007.928648:0:42:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:6.0:1584542007.928649:0:42:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:6.0:1584542007.928651:0:42:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23807 00000100:00080000:6.0:1584542007.928653:0:42:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:6.0:1584542007.928658:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584542007.928659:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584542007.928661:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584542007.928662:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584542007.928663:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584542007.928664:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:6.0:1584542007.928664:0:42:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:6.0:1584542007.928665:0:42:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542007.978508:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584542007.978512:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584542007.978515:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584542007.978516:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584542007.978517:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584542007.978519:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584542007.978520:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584542007.978522:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542088 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:6.0:1584542007.978551:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078466656 new x1661492078466720 00000800:00020000:8.0:1584542013.916066:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584542013.916069:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584542013.916071:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542013.916686:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584542013.928762:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542007/real 1584542007] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542094 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584542013.928775:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584542013.928776:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584542013.928778:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584542013.928784:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584542013.928785:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584542013.928786:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584542013.928787:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584542013.928788:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584542013.928789:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584542013.928790:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584542013.928792:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23813 00000100:00080000:8.0:1584542013.928794:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584542013.928806:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584542013.928807:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584542013.928808:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584542013.928809:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584542013.928809:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584542013.928811:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584542013.928811:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584542013.928812:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542013.967213:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584542013.967218:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584542013.967219:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584542013.967221:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584542013.967222:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584542013.967225:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584542013.967226:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584542013.967228:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542094 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584542013.967238:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078466720 new x1661492078466768 00000800:00000100:0.0:1584542020.143226:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584542020.143654:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584542020.155702:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542013/real 1584542013] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542100 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584542020.199475:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584542020.199476:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584542020.199477:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584542020.199481:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584542020.199483:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584542020.199483:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584542020.199485:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584542020.199485:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584542020.199487:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584542020.199487:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584542020.199489:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23819 00000100:00080000:8.0:1584542020.199491:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584542020.199499:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584542020.199500:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584542020.199501:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584542020.199502:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584542020.199502:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584542020.199503:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584542020.199505:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584542020.199506:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542020.245126:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584542020.245130:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584542020.245132:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584542020.245133:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584542020.245134:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584542020.245136:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584542020.245137:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584542020.245139:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542100 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584542020.245168:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078466768 new x1661492078466800 00000800:00020000:2.0:1584542025.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542025.916077:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542025.916079:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542025.916512:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:8.0:1584542025.928589:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542020/real 1584542020] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542107 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:8.0:1584542025.928596:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:8.0:1584542025.928597:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:8.0:1584542025.928599:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:8.0:1584542025.928604:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584542025.928605:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584542025.928606:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584542025.928607:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584542025.928608:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584542025.928609:0:171:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:8.0:1584542025.928610:0:171:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:8.0:1584542025.928611:0:171:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23826 00000100:00080000:8.0:1584542025.928613:0:171:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:8.0:1584542025.928621:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584542025.928622:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584542025.928623:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584542025.928624:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584542025.928625:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584542025.928627:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:8.0:1584542025.928627:0:171:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:8.0:1584542025.928628:0:171:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542025.949371:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584542025.949375:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584542025.949377:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584542025.949378:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584542025.949379:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584542025.949381:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584542025.949382:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584542025.949384:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542107 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:8.0:1584542025.949415:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078466800 new x1661492078466832 00000800:00020000:8.0:1584542031.916072:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584542031.916074:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584542031.916076:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542031.916522:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584542031.928578:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542025/real 1584542025] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542112 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584542031.928586:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584542031.928587:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584542031.928588:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584542031.928604:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584542031.928605:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584542031.928608:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584542031.928610:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584542031.928610:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584542031.928612:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584542031.928612:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584542031.928614:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23831 00000100:00080000:10.0:1584542031.928616:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584542031.928622:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584542031.928624:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584542031.928624:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584542031.928625:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584542031.928627:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584542031.928628:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584542031.928629:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584542031.928630:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542031.978535:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584542031.978539:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584542031.978540:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584542031.978541:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584542031.978542:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584542031.978544:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584542031.978546:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584542031.978548:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542112 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584542031.978579:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078466832 new x1661492078466912 00000800:00020000:2.0:1584542037.916113:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542037.916115:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542037.916117:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542037.916626:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584542037.928674:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542031/real 1584542031] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542118 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584542037.928681:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584542037.928683:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584542037.928684:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584542037.928689:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584542037.928691:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584542037.928691:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584542037.928692:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584542037.928693:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584542037.928695:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584542037.928695:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584542037.928697:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23837 00000100:00080000:10.0:1584542037.928699:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584542037.928707:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584542037.928708:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584542037.928710:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584542037.928711:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584542037.928712:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584542037.928713:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584542037.928713:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584542037.928714:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542037.986381:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584542037.986385:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584542037.986386:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584542037.986387:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584542037.986388:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584542037.986390:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584542037.986391:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584542037.986393:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542118 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:10.0:1584542037.986422:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078466912 new x1661492078466944 00000800:00000100:0.0:1584542044.143794:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584542044.144225:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:10.0:1584542044.156296:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542037/real 1584542037] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542124 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:10.0:1584542044.156304:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:10.0:1584542044.181656:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:10.0:1584542044.181658:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:10.0:1584542044.181663:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584542044.181665:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584542044.181666:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584542044.181667:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584542044.181667:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584542044.181669:0:334:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:10.0:1584542044.181670:0:334:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:10.0:1584542044.181672:0:334:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23843 00000100:00080000:10.0:1584542044.181674:0:334:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:10.0:1584542044.181684:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584542044.181686:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584542044.181687:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584542044.181688:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584542044.181688:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584542044.181690:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:10.0:1584542044.181690:0:334:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:10.0:1584542044.181691:0:334:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542044.230309:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584542044.230314:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584542044.230317:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584542044.230318:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584542044.230320:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584542044.230322:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584542044.230323:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584542044.248114:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542124 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584542044.248123:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078466944 new x1661492078466992 00000800:00020000:2.0:1584542049.916063:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542049.916067:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542049.916069:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542049.916499:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584542049.928609:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542044/real 1584542044] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542131 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584542049.928621:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584542049.928622:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584542049.928624:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584542049.928631:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542049.928633:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542049.928634:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542049.928635:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542049.928636:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542049.928637:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584542049.928638:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584542049.928640:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23850 00000100:00080000:12.0:1584542049.928643:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584542049.928659:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542049.928660:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542049.928661:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542049.928662:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542049.928663:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542049.928664:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542049.928665:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542049.928666:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542049.973596:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:16.0:1584542049.973602:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:16.0:1584542049.973604:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:16.0:1584542049.973606:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:16.0:1584542049.973607:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:16.0:1584542049.973610:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:16.0:1584542049.973611:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:16.0:1584542049.973613:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542131 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584542049.973643:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078466992 new x1661492078467024 00000800:00020000:2.0:1584542055.916049:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542055.916053:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542055.916055:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542055.916516:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584542055.928568:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542049/real 1584542049] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542136 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584542055.928578:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584542055.928580:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584542055.928582:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584542055.928603:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542055.928605:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542055.928614:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542055.928616:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542055.928617:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542055.928618:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584542055.928619:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584542055.928622:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23855 00000100:00080000:12.0:1584542055.928625:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584542055.928632:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542055.928634:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542055.928635:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542055.928636:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542055.928637:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542055.928638:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542055.928640:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542055.928641:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542055.946222:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584542055.946229:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584542055.946232:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584542055.946233:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584542055.946235:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584542055.946237:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584542055.946238:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584542055.946240:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542136 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584542055.946270:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078467024 new x1661492078467088 00000800:00020000:6.0:1584542061.916043:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584542061.916046:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584542061.916048:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542061.916461:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584542061.928510:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542055/real 1584542055] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542142 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584542061.928518:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584542061.928520:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584542061.928521:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584542061.928526:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542061.928528:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542061.928529:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542061.928531:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542061.928531:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542061.928533:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584542061.928533:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584542061.928535:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23861 00000100:00080000:12.0:1584542061.928538:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584542061.928548:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542061.928549:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542061.928550:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542061.928551:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542061.928553:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542061.928554:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542061.928555:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542061.928556:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542061.981123:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584542061.981131:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584542061.981133:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584542061.981134:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584542061.981135:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584542061.981138:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584542061.981139:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584542061.981141:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542142 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584542061.981151:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078467088 new x1661492078467136 00000800:00000100:0.0:1584542068.139392:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584542068.139822:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584542068.151893:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542061/real 1584542061] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542148 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584542068.151902:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584542068.151903:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584542068.151905:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584542068.151912:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542068.151914:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542068.151914:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542068.151916:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542068.151916:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542068.151918:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584542068.151919:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584542068.151920:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23867 00000100:00080000:12.0:1584542068.151923:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584542068.151934:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542068.151935:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542068.151937:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542068.151939:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542068.151940:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542068.151941:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542068.151941:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542068.151943:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542068.205148:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584542068.205154:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584542068.205156:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584542068.205157:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584542068.205159:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584542068.205161:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584542068.205163:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584542068.205165:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542148 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584542068.205174:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078467136 new x1661492078467168 00000400:00000080:2.0:1584542072.786659:0:183080:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00020000:2.0:1584542073.916092:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542073.916094:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542073.916096:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542073.916692:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584542073.928746:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542068/real 1584542068] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542155 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584542073.928754:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584542073.928756:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584542073.928757:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584542073.928763:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542073.928765:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542073.928766:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542073.928767:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542073.928767:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542073.928769:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584542073.928770:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584542073.928771:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23874 00000100:00080000:12.0:1584542073.928773:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584542073.928783:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542073.928785:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542073.928785:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542073.928787:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542073.928788:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542073.928789:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542073.928789:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542073.928791:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000400:00000080:3.0:1584542073.956003:0:183082:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000100:00080000:18.0:1584542073.984828:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584542073.984833:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584542073.984834:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584542073.984835:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584542073.984837:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584542073.984839:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584542073.984840:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584542073.984842:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542155 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584542073.984871:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078467168 new x1661492078467216 00000400:00000080:2.0:1584542074.719432:0:183085:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584542075.400326:0:183088:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584542076.033426:0:183089:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584542076.664395:0:183092:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584542077.303496:0:183094:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584542077.889689:0:183096:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:3.0:1584542078.535536:0:183098:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000400:00000080:2.0:1584542079.120378:0:183100:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775677 00000800:00020000:2.0:1584542079.916099:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542079.916101:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542079.916103:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542079.916566:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584542079.928619:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542073/real 1584542073] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542160 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584542079.928627:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584542079.928628:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584542079.928629:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584542079.928643:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542079.928645:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542079.928653:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542079.928654:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542079.928654:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542079.928656:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584542079.928656:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584542079.928659:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23879 00000100:00080000:12.0:1584542079.928661:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584542079.928666:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542079.928667:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542079.928668:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542079.928669:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542079.928670:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542079.928671:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542079.928671:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542079.928672:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542079.949850:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584542079.949855:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584542079.949856:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584542079.949857:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584542079.949858:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584542079.949860:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584542079.949861:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584542079.949863:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542160 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584542079.949892:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078467216 new x1661492078467280 00000800:00020000:8.0:1584542085.916063:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584542085.916066:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584542085.916068:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542085.916448:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584542085.928503:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542079/real 1584542079] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542166 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584542085.928511:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584542085.928512:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584542085.928513:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584542085.928519:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542085.928520:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542085.928521:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542085.928522:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542085.928523:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542085.928525:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584542085.928526:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584542085.928527:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23885 00000100:00080000:12.0:1584542085.928529:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584542085.928538:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542085.928539:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542085.928539:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542085.928541:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542085.928541:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542085.928542:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542085.928543:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542085.928544:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542085.974376:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584542085.974380:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584542085.974382:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584542085.974383:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584542085.974384:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584542085.974386:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584542085.974387:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584542085.974389:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542166 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584542085.974418:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078467280 new x1661492078467312 00000800:00000100:0.0:1584542092.139963:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584542092.140429:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584542092.152480:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542085/real 1584542085] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542172 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584542092.152488:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584542092.152489:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584542092.152490:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584542092.152496:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542092.152498:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542092.152499:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542092.152501:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542092.152501:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542092.152503:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584542092.152503:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584542092.152505:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23891 00000100:00080000:12.0:1584542092.152507:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584542092.152516:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542092.152517:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542092.152517:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542092.152519:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542092.152519:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542092.152520:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542092.152521:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542092.152522:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542092.198048:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584542092.198052:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584542092.198054:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584542092.198055:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584542092.198056:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584542092.198058:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584542092.198059:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584542092.198061:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542172 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584542092.198090:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078467312 new x1661492078467360 00000800:00020000:2.0:1584542097.916080:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542097.916083:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542097.916085:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542097.916556:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584542097.928601:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542092/real 1584542092] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542179 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584542097.928609:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584542097.928610:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584542097.928611:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584542097.928617:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542097.928619:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542097.928620:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542097.928621:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542097.928622:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542097.928623:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584542097.928624:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584542097.928626:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23898 00000100:00080000:12.0:1584542097.928627:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584542097.928635:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542097.928637:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542097.928637:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542097.928638:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542097.928639:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542097.928640:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542097.928641:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542097.928642:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542097.978382:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584542097.978387:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584542097.978388:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584542097.978389:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584542097.978391:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584542097.978392:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584542097.978394:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584542097.978396:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542179 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:12.0:1584542097.978424:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078467360 new x1661492078467392 00000800:00020000:8.0:1584542103.916062:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584542103.916065:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584542103.916067:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542103.916494:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:12.0:1584542103.928552:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542097/real 1584542097] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542184 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:12.0:1584542103.928563:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:12.0:1584542103.928564:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:12.0:1584542103.928568:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:12.0:1584542103.928589:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542103.928591:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542103.928595:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542103.928596:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542103.928597:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542103.928599:0:4059:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:12.0:1584542103.928600:0:4059:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:12.0:1584542103.928602:0:4059:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23903 00000100:00080000:12.0:1584542103.928606:0:4059:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:12.0:1584542103.928613:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542103.928615:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542103.928616:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542103.928617:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542103.928617:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542103.928619:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:12.0:1584542103.928619:0:4059:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:12.0:1584542103.928621:0:4059:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542103.946387:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584542103.946394:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584542103.946397:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584542103.946398:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584542103.946400:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584542103.946402:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584542103.946404:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584542103.946406:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542184 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:14.0:1584542103.946435:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078467392 new x1661492078467472 00000800:00020000:2.0:1584542109.916140:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542109.916144:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542109.916146:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542109.916671:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584542109.928759:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542103/real 1584542103] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542190 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584542109.928770:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584542109.928771:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584542109.928774:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584542109.928781:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542109.928783:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542109.928784:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542109.928785:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542109.928787:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542109.928789:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584542109.928790:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584542109.928792:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23909 00000100:00080000:16.0:1584542109.928795:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584542109.928813:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542109.928815:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542109.928815:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542109.928817:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542109.928817:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542109.928819:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542109.928819:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542109.928821:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542109.987974:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584542109.987981:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584542109.987983:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584542109.987984:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584542109.987986:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584542109.987988:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584542109.987990:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584542109.987992:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542190 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584542109.988022:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078467472 new x1661492078467504 00000800:00000100:0.0:1584542116.141115:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584542116.141580:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584542116.153638:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542109/real 1584542109] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542196 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584542116.153647:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584542116.153648:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584542116.153650:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584542116.153657:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542116.153659:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542116.153661:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542116.153662:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542116.153663:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542116.153665:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584542116.153665:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584542116.153667:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23915 00000100:00080000:16.0:1584542116.153670:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584542116.153685:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542116.153686:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542116.153687:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542116.153688:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542116.153688:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542116.153690:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542116.153691:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542116.153692:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542116.208591:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584542116.208599:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584542116.208601:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584542116.208602:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584542116.208604:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584542116.208606:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584542116.208608:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584542116.208610:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542196 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584542116.208639:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078467504 new x1661492078467536 00000800:00020000:8.0:1584542121.916129:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584542121.916132:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584542121.916134:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542121.916551:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584542121.928604:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542116/real 1584542116] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542203 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584542121.928612:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584542121.928613:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584542121.928615:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584542121.928622:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542121.928624:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542121.928625:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542121.928626:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542121.928626:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542121.928628:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584542121.928629:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584542121.928631:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23922 00000100:00080000:16.0:1584542121.928633:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584542121.928643:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542121.928644:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542121.928645:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542121.928646:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542121.928647:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542121.928648:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542121.928649:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542121.928650:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542121.964079:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:18.0:1584542121.964084:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:18.0:1584542121.964086:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:18.0:1584542121.964087:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:18.0:1584542121.964088:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:18.0:1584542121.964090:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:18.0:1584542121.964092:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:18.0:1584542121.964093:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542203 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584542121.964123:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078467536 new x1661492078467584 00000800:00020000:2.0:1584542127.916070:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542127.916073:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542127.916074:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542127.916516:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584542127.928567:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542121/real 1584542121] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542208 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584542127.928575:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584542127.928577:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584542127.928579:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584542127.928593:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542127.928595:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542127.928600:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542127.928601:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542127.928602:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542127.928603:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584542127.928604:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584542127.928606:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23927 00000100:00080000:16.0:1584542127.928608:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584542127.928613:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542127.928614:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542127.928615:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542127.928616:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542127.928617:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542127.928618:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542127.928619:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542127.928620:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584542127.943271:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584542127.943276:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584542127.943277:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584542127.943278:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584542127.943279:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584542127.943281:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584542127.943283:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584542127.943285:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542208 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584542127.943314:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078467584 new x1661492078467648 00000800:00020000:4.0:1584542133.916046:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584542133.916049:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584542133.916051:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542133.916454:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584542133.928507:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542127/real 1584542127] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542214 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584542133.928516:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584542133.928517:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584542133.928518:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584542133.928525:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542133.928526:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542133.928527:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542133.928528:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542133.928529:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542133.928530:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584542133.928531:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584542133.928533:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23933 00000100:00080000:16.0:1584542133.928535:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584542133.928543:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542133.928544:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542133.928545:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542133.928546:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542133.928547:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542133.928548:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542133.928549:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542133.928550:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584542133.971744:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584542133.971748:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584542133.971749:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584542133.971750:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584542133.971751:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584542133.971753:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584542133.971755:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584542133.971757:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542214 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584542133.971787:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078467648 new x1661492078467696 00000800:00000100:0.0:1584542140.141610:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584542140.142024:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584542140.154076:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542133/real 1584542133] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542220 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584542140.154084:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584542140.154085:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584542140.154086:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584542140.154092:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542140.154093:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542140.154094:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542140.154095:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542140.154096:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542140.154097:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584542140.154098:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584542140.154099:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23939 00000100:00080000:16.0:1584542140.154101:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584542140.154110:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542140.154111:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542140.154112:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542140.154113:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542140.154114:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542140.154115:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542140.154116:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542140.154118:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584542140.200453:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584542140.200457:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584542140.200459:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584542140.200459:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584542140.200461:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584542140.200462:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584542140.200464:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584542140.200466:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542220 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584542140.200475:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078467696 new x1661492078467728 00000800:00020000:2.0:1584542145.916059:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542145.916062:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542145.916064:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542145.916632:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584542145.928693:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542140/real 1584542140] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542227 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584542145.928701:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584542145.928702:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584542145.928703:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584542145.928709:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542145.928710:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542145.928711:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542145.928712:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542145.928713:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542145.928714:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584542145.928715:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584542145.928717:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23946 00000100:00080000:16.0:1584542145.928719:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584542145.928727:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542145.928728:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542145.928729:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542145.928730:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542145.928731:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542145.928733:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542145.928733:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542145.928734:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:20.0:1584542145.972372:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:20.0:1584542145.972376:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:20.0:1584542145.972377:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:20.0:1584542145.972378:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:20.0:1584542145.972379:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:20.0:1584542145.972381:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:20.0:1584542145.972384:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:20.0:1584542145.972386:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542227 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584542145.972416:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078467728 new x1661492078467760 00000800:00020000:0.0:1584542151.916065:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584542151.916068:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584542151.916070:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542151.916449:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584542151.928500:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542145/real 1584542145] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542232 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584542151.928508:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584542151.928509:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584542151.928510:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584542151.928525:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542151.928526:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542151.928529:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542151.928530:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542151.928531:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542151.928532:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584542151.928533:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584542151.928535:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23951 00000100:00080000:16.0:1584542151.928537:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584542151.928542:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542151.928544:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542151.928545:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542151.928546:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542151.928547:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542151.928548:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542151.928549:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542151.928550:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542151.983021:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542151.983049:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542151.983051:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542151.983051:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542151.983053:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542151.983055:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542151.983057:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542151.983058:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542232 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584542151.983088:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078467760 new x1661492078467840 00000800:00020000:4.0:1584542157.916057:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584542157.916060:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584542157.916062:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542157.916473:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584542157.928525:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542151/real 1584542151] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542238 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584542157.928533:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584542157.928534:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584542157.928536:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584542157.928541:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542157.928543:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542157.928544:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542157.928545:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542157.928545:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542157.928547:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584542157.928547:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584542157.928549:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23957 00000100:00080000:16.0:1584542157.928551:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584542157.928560:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542157.928561:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542157.928562:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542157.928563:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542157.928564:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542157.928565:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542157.928565:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542157.928567:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542157.972195:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542157.972199:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542157.972200:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542157.972201:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542157.972203:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542157.972205:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542157.972206:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542157.972208:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542238 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584542157.972242:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078467840 new x1661492078467872 00000800:00000100:0.0:1584542164.142149:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584542164.142625:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584542164.154682:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542157/real 1584542157] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542244 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584542164.154692:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584542164.154693:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584542164.154696:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584542164.154703:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542164.154705:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542164.154706:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542164.154708:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542164.154708:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542164.154710:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584542164.154711:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584542164.154714:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23963 00000100:00080000:16.0:1584542164.154717:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584542164.154731:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542164.154733:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542164.154733:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542164.154734:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542164.154735:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542164.154736:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542164.154737:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542164.154738:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542164.198649:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542164.198655:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542164.198657:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542164.198658:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542164.198660:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542164.198663:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542164.198664:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542164.198666:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542244 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584542164.198696:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078467872 new x1661492078467920 00000800:00020000:4.0:1584542169.916038:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584542169.916041:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584542169.916043:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542169.916426:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584542169.928475:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542164/real 1584542164] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542251 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584542169.928483:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584542169.928485:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584542169.928486:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584542169.928492:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542169.928494:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542169.928495:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542169.928496:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542169.928497:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542169.928499:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584542169.928500:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584542169.928502:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23970 00000100:00080000:16.0:1584542169.928504:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584542169.928516:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542169.928518:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542169.928518:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542169.928520:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542169.928520:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542169.928521:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542169.928522:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542169.928523:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542169.962606:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542169.962611:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542169.962613:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542169.962615:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542169.962616:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542169.962618:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542169.962620:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542169.962622:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542251 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584542169.962652:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078467920 new x1661492078467952 00000800:00020000:2.0:1584542175.916052:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542175.916055:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542175.916057:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542175.916479:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584542175.928536:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542169/real 1584542169] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542256 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584542175.928545:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584542175.928547:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584542175.928548:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584542175.928568:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542175.928570:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542175.928574:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542175.928576:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542175.928577:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542175.928579:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584542175.928580:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584542175.928582:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23975 00000100:00080000:16.0:1584542175.928584:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584542175.928591:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542175.928592:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542175.928593:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542175.928594:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542175.928595:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542175.928597:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542175.928597:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542175.928598:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542175.974496:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542175.974502:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542175.974504:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542175.974505:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542175.974507:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542175.974509:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542175.974511:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542175.974513:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542256 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584542175.974548:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078467952 new x1661492078468016 00000800:00020000:4.0:1584542181.916043:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584542181.916047:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584542181.916048:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542181.916497:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584542181.928554:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542175/real 1584542175] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542262 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584542181.928563:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584542181.928564:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584542181.928566:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584542181.928573:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542181.928575:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542181.928576:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542181.928577:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542181.928578:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542181.928580:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584542181.928580:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584542181.928582:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23981 00000100:00080000:16.0:1584542181.928585:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584542181.928596:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542181.928597:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542181.928598:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542181.928599:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542181.928600:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542181.928601:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542181.928601:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542181.928603:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542181.978536:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542181.978541:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542181.978543:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542181.978545:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542181.978546:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542181.978549:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542181.978550:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542181.978553:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542262 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584542181.978587:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078468016 new x1661492078468064 00000800:00000100:0.0:1584542188.140379:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584542188.140799:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584542188.152850:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542181/real 1584542181] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542268 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584542188.152858:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584542188.152859:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584542188.152861:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584542188.152867:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542188.152868:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542188.152869:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542188.152870:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542188.152871:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542188.152872:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584542188.152873:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584542188.152874:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23987 00000100:00080000:16.0:1584542188.152876:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584542188.152885:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542188.152886:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542188.152887:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542188.152888:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542188.152889:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542188.152890:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542188.152890:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542188.152892:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542188.201377:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542188.201381:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542188.201382:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542188.201383:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542188.201385:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542188.201387:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542188.201388:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542188.201390:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542268 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584542188.201420:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078468064 new x1661492078468096 00000800:00020000:8.0:1584542193.916071:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:8.0:1584542193.916074:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:8.0:1584542193.916076:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542193.916576:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584542193.928630:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542188/real 1584542188] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584542193.928643:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584542193.928644:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584542193.928645:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584542193.928651:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542193.928653:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542193.928653:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542193.928654:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542193.928655:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542193.928657:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584542193.928657:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584542193.928659:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23994 00000100:00080000:16.0:1584542193.928661:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584542193.928669:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542193.928671:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542193.928671:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542193.928672:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542193.928673:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542193.928674:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542193.928675:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542193.928676:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542193.947162:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542193.947166:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542193.947168:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542193.947169:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542193.947170:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542193.947172:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542193.947174:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542193.947175:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584542193.947205:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078468096 new x1661492078468144 00000800:00020000:4.0:1584542199.916084:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584542199.916087:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584542199.916089:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542199.916566:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584542199.928627:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542193/real 1584542193] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584542199.928635:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584542199.928636:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584542199.928637:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584542199.928653:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542199.928654:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542199.928657:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542199.928659:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542199.928659:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542199.928661:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584542199.928661:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584542199.928663:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 23999 00000100:00080000:16.0:1584542199.928665:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584542199.928670:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542199.928672:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542199.928672:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542199.928673:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542199.928674:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542199.928675:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542199.928677:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542199.928678:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542199.979448:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542199.979452:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542199.979454:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542199.979454:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542199.979456:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542199.979457:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542199.979459:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542199.979461:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584542199.979490:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078468144 new x1661492078468208 00000800:00020000:2.0:1584542205.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542205.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542205.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542205.916520:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584542205.928573:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542199/real 1584542199] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584542205.928581:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584542205.928582:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584542205.928584:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584542205.928589:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542205.928591:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542205.928592:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542205.928593:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542205.928593:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542205.928595:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584542205.928595:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584542205.928597:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 24005 00000100:00080000:16.0:1584542205.928599:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584542205.928608:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542205.928609:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542205.928610:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542205.928611:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542205.928613:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542205.928614:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542205.928614:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542205.928616:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542205.987713:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542205.987717:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542205.987719:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542205.987720:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542205.987721:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542205.987723:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542205.987724:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542205.987726:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584542205.987737:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078468208 new x1661492078468240 00000800:00000100:0.0:1584542212.141466:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584542212.141902:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584542212.153944:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542205/real 1584542205] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584542212.153952:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584542212.153953:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584542212.153954:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584542212.153960:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542212.153961:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542212.153962:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542212.153963:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542212.153964:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542212.153965:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584542212.153966:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584542212.153968:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 24011 00000100:00080000:16.0:1584542212.153970:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584542212.153978:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542212.153979:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542212.153981:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542212.153982:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542212.153982:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542212.153984:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542212.153984:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542212.153985:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542212.201024:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542212.201057:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542212.201059:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542212.201060:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542212.201061:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542212.201063:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542212.201065:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542212.201067:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584542212.201078:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078468240 new x1661492078468288 00000800:00020000:2.0:1584542217.916088:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542217.916091:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542217.916094:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542217.916608:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584542217.928666:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542212/real 1584542212] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584542217.928677:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584542217.928678:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584542217.928680:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584542217.928688:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542217.928690:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542217.928691:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542217.928692:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542217.928693:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542217.928694:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584542217.928695:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584542217.928697:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 24018 00000100:00080000:16.0:1584542217.928700:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584542217.928717:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542217.928718:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542217.928719:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542217.928720:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542217.928720:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542217.928722:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542217.928722:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542217.928724:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542217.981494:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542217.981500:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542217.981503:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542217.981504:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542217.981506:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542217.981509:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542217.981511:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542217.981512:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584542217.981542:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078468288 new x1661492078468320 00000800:00020000:6.0:1584542223.916064:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:6.0:1584542223.916067:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:6.0:1584542223.916070:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542223.916535:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584542223.928648:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542217/real 1584542217] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584542223.928659:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584542223.928661:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584542223.928663:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584542223.928686:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542223.928689:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542223.928704:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542223.928705:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542223.928706:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542223.928707:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584542223.928708:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584542223.928712:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 24023 00000100:00080000:16.0:1584542223.928716:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584542223.928725:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542223.928726:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542223.928727:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542223.928728:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542223.928729:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542223.928731:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542223.928731:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542223.928733:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542223.977317:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542223.977324:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542223.977326:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542223.977328:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542223.977330:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542223.977332:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542223.977334:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542223.977336:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584542223.977346:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078468320 new x1661492078468400 00000800:00020000:4.0:1584542229.916082:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584542229.916086:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584542229.916089:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542229.916553:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584542229.928674:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542223/real 1584542223] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584542229.928685:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584542229.928687:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584542229.928689:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584542229.928698:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542229.928700:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542229.928701:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542229.928702:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542229.928703:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542229.928706:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584542229.928707:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584542229.928709:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 24029 00000100:00080000:16.0:1584542229.928712:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584542229.928729:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542229.928730:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542229.928731:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542229.928732:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542229.928733:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542229.928735:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542229.928735:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542229.928737:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542229.950719:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542229.950726:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542229.950728:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542229.950729:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542229.950731:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542229.950734:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542229.950735:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542229.950737:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584542229.950767:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078468400 new x1661492078468432 00000800:00000100:0.0:1584542236.140611:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584542236.141041:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584542236.153094:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542229/real 1584542229] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584542236.153102:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584542236.153103:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584542236.153104:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584542236.153110:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542236.153112:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542236.153113:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542236.153115:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542236.153116:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542236.153117:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584542236.153118:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584542236.153120:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 24035 00000100:00080000:16.0:1584542236.153122:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584542236.153133:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542236.153134:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542236.153135:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542236.153136:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542236.153136:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542236.153138:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542236.153138:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542236.153140:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542236.210796:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542236.210801:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542236.210803:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542236.210804:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542236.210805:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542236.210807:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542236.210809:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542236.210810:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584542236.210820:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078468432 new x1661492078468464 00000800:00020000:2.0:1584542241.916071:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542241.916074:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542241.916076:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542241.916465:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584542241.928525:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542236/real 1584542236] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542323 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584542241.928534:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584542241.928536:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584542241.928537:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584542241.928545:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542241.928547:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542241.928549:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542241.928550:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542241.928551:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542241.928552:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584542241.928553:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584542241.928555:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 24042 00000100:00080000:16.0:1584542241.928557:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584542241.928569:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542241.928570:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542241.928571:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542241.928572:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542241.928573:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542241.928574:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542241.928575:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542241.928576:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542241.975839:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542241.975845:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542241.975847:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542241.975849:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542241.975850:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542241.975853:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542241.975854:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542241.975856:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542323 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:16.0:1584542241.975886:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078468464 new x1661492078468512 00000800:00020000:2.0:1584542247.916062:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542247.916065:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542247.916067:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542247.916477:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:16.0:1584542247.928519:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542241/real 1584542241] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542328 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:16.0:1584542247.928527:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:16.0:1584542247.928528:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:16.0:1584542247.928530:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:16.0:1584542247.928544:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542247.928546:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542247.928549:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542247.928550:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542247.928551:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542247.928552:0:4060:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:16.0:1584542247.928553:0:4060:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:16.0:1584542247.928554:0:4060:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 24047 00000100:00080000:16.0:1584542247.928557:0:4060:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:16.0:1584542247.928562:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542247.928563:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542247.928564:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542247.928565:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542247.928565:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542247.928567:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:16.0:1584542247.928567:0:4060:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:16.0:1584542247.928568:0:4060:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542247.984038:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542247.984042:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542247.984043:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542247.984044:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542247.984045:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542247.984047:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542247.984049:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542247.984050:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542328 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584542247.984081:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078468512 new x1661492078468576 00000800:00020000:2.0:1584542253.916060:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542253.916063:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542253.916065:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542253.916477:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584542253.928555:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542247/real 1584542247] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542334 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584542253.928564:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584542253.928565:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584542253.928566:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584542253.928573:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542253.928574:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542253.928576:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542253.928577:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542253.928578:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542253.928579:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584542253.928580:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584542253.928582:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 24053 00000100:00080000:18.0:1584542253.928584:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584542253.928594:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542253.928595:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542253.928596:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542253.928597:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542253.928598:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542253.928599:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542253.928600:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542253.928601:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542253.955142:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542253.955146:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542253.955148:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542253.955149:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542253.955150:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542253.955152:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542253.955154:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542253.955155:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542334 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584542253.955184:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078468576 new x1661492078468624 00000800:00000100:0.0:1584542260.141116:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584542260.141573:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584542260.153640:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542253/real 1584542253] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542340 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584542260.153648:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584542260.153649:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584542260.153651:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584542260.153657:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542260.153659:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542260.153660:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542260.153661:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542260.153661:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542260.153663:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584542260.153663:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584542260.153665:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 24059 00000100:00080000:18.0:1584542260.153667:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584542260.153676:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542260.153677:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542260.153678:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542260.153679:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542260.153680:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542260.153681:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542260.153681:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542260.153683:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542260.208854:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542260.208858:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542260.208859:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542260.208860:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542260.208861:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542260.208863:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542260.208865:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542260.208866:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542340 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584542260.208896:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078468624 new x1661492078468656 00000800:00020000:2.0:1584542265.916058:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542265.916061:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542265.916062:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542265.916496:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584542265.928566:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542260/real 1584542260] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542347 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584542265.928574:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584542265.928576:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584542265.928577:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584542265.928583:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542265.928585:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542265.928585:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542265.928587:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542265.928587:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542265.928589:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584542265.928589:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584542265.928591:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 24066 00000100:00080000:18.0:1584542265.928593:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584542265.928601:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542265.928603:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542265.928603:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542265.928604:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542265.928605:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542265.928606:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542265.928607:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542265.928608:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542265.951816:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542265.951820:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542265.951822:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542265.951823:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542265.951824:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542265.951826:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542265.951827:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542265.951829:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542347 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584542265.951859:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078468656 new x1661492078468688 00000800:00020000:2.0:1584542271.916060:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542271.916063:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542271.916065:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542271.916507:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584542271.928585:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542265/real 1584542265] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542352 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584542271.928594:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584542271.928595:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584542271.928596:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584542271.928612:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542271.928614:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542271.928617:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542271.928618:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542271.928619:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542271.928620:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584542271.928621:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584542271.928622:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 24071 00000100:00080000:18.0:1584542271.928624:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584542271.928630:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542271.928631:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542271.928632:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542271.928633:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542271.928633:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542271.928634:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542271.928635:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542271.928636:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542271.970388:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542271.970392:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542271.970393:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542271.970394:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542271.970395:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542271.970397:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542271.970399:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542271.970401:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542352 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584542271.970430:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078468688 new x1661492078468768 00000800:00020000:4.0:1584542277.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584542277.916081:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584542277.916083:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542277.916484:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584542277.928561:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542271/real 1584542271] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542358 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584542277.928569:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584542277.928570:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584542277.928571:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584542277.928577:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542277.928578:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542277.928579:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542277.928580:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542277.928581:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542277.928582:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584542277.928583:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584542277.928585:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 24077 00000100:00080000:18.0:1584542277.928587:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584542277.928595:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542277.928597:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542277.928597:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542277.928598:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542277.928599:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542277.928600:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542277.928602:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542277.928603:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542277.977290:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542277.977294:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542277.977295:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542277.977296:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542277.977298:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542277.977299:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542277.977301:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542277.977303:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542358 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584542277.977332:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078468768 new x1661492078468800 00000800:00000100:0.0:1584542284.141749:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584542284.142147:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584542284.154216:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542277/real 1584542277] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542364 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584542284.154225:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584542284.154226:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584542284.154228:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584542284.154234:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542284.154236:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542284.154237:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542284.154238:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542284.154239:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542284.154240:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584542284.154241:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584542284.154243:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 24083 00000100:00080000:18.0:1584542284.154246:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584542284.154256:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542284.154257:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542284.154258:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542284.154259:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542284.154261:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542284.154262:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542284.154262:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542284.154264:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542284.198046:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542284.198052:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542284.198054:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542284.198055:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542284.198057:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542284.198059:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542284.198061:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542284.198063:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542364 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584542284.198094:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078468800 new x1661492078468848 00000800:00020000:4.0:1584542289.916101:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584542289.916105:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584542289.916107:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542289.916548:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584542289.928636:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542284/real 1584542284] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542371 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584542289.928647:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584542289.928649:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584542289.928650:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584542289.928659:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542289.928661:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542289.928662:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542289.928663:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542289.928664:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542289.928665:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584542289.928666:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584542289.928668:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 24090 00000100:00080000:18.0:1584542289.928672:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584542289.928688:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542289.928689:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542289.928692:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542289.928693:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542289.928694:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542289.928695:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542289.928696:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542289.928697:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542289.982286:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542289.982293:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542289.982295:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542289.982296:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542289.982298:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542289.982300:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542289.982302:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542289.982304:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542371 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584542289.982335:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078468848 new x1661492078468880 00000800:00020000:2.0:1584542295.916039:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542295.916043:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542295.916046:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542295.916458:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584542295.928538:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542289/real 1584542289] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542376 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584542295.928548:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584542295.928550:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584542295.928552:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584542295.928576:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542295.928578:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542295.928582:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542295.928583:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542295.928584:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542295.928586:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584542295.928587:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584542295.928589:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 24095 00000100:00080000:18.0:1584542295.928592:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584542295.928601:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542295.928602:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542295.928603:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542295.928604:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542295.928605:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542295.928606:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542295.928607:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542295.928608:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542295.978171:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542295.978178:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542295.978180:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542295.978182:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542295.978183:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542295.978186:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542295.978188:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542295.978190:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542376 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584542295.978219:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078468880 new x1661492078468944 00000800:00020000:2.0:1584542301.916041:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542301.916043:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542301.916045:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542301.916507:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584542301.928581:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542295/real 1584542295] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542382 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584542301.928590:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584542301.928592:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584542301.928593:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584542301.928600:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542301.928602:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542301.928603:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542301.928604:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542301.928604:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542301.928606:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584542301.928607:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584542301.928610:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 24101 00000100:00080000:18.0:1584542301.928613:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584542301.928624:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542301.928626:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542301.928626:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542301.928628:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542301.928628:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542301.928630:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542301.928630:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542301.928632:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542301.966369:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542301.966375:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542301.966377:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542301.966378:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542301.966380:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542301.966382:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542301.966384:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542301.966385:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542382 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584542301.966416:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078468944 new x1661492078468992 00000800:00000100:0.0:1584542308.135690:0:166:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error 0(waiting) 00000100:00020000:0.0:1584542308.136123:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584542308.148191:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542301/real 1584542301] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542388 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584542308.148199:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584542308.148200:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584542308.148202:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584542308.148208:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542308.148210:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542308.148211:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542308.148212:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542308.148213:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542308.148215:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584542308.148216:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584542308.148218:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 24107 00000100:00080000:18.0:1584542308.148220:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584542308.148233:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542308.148234:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542308.148235:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542308.148236:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542308.148237:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542308.148238:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542308.148239:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542308.148240:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542308.186985:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542308.186991:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542308.186993:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542308.186994:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542308.186995:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542308.186997:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542308.186999:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542308.187001:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542388 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584542308.187079:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078468992 new x1661492078469024 00000800:00020000:4.0:1584542313.916078:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584542313.916081:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584542313.916083:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542313.916497:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584542313.928576:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542308/real 1584542308] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542395 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584542313.928585:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584542313.928586:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584542313.928587:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584542313.928594:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542313.928596:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542313.928596:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542313.928599:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542313.928599:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542313.928601:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584542313.928602:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584542313.928603:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 24114 00000100:00080000:18.0:1584542313.928606:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584542313.928617:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542313.928619:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542313.928619:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542313.928621:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542313.928621:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542313.928623:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542313.928623:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542313.928624:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542313.935392:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542313.935397:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542313.935399:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542313.935400:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542313.935401:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542313.935404:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542313.935405:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542313.935407:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542395 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584542313.935436:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078469024 new x1661492078469072 00000800:00020000:4.0:1584542319.916069:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:4.0:1584542319.916072:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:4.0:1584542319.916074:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542319.916486:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584542319.928539:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542313/real 1584542313] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542400 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584542319.928547:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584542319.928548:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584542319.928549:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584542319.928565:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542319.928567:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542319.928570:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542319.928572:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542319.928572:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542319.928574:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584542319.928574:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584542319.928576:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 24119 00000100:00080000:18.0:1584542319.928578:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584542319.928583:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542319.928585:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542319.928585:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542319.928586:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542319.928587:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542319.928588:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542319.928589:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542319.928590:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542319.934986:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542319.934990:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542319.934992:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542319.934993:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542319.934994:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542319.934996:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542319.934997:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542319.934999:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542400 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584542319.935062:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078469072 new x1661492078469136 00000800:00020000:0.0:1584542325.916075:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:0.0:1584542325.916078:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:0.0:1584542325.916080:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542325.916495:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584542325.928544:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542319/real 1584542319] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542406 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584542325.928552:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584542325.928553:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584542325.928555:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584542325.928561:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542325.928562:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542325.928563:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542325.928565:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542325.928565:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542325.928567:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584542325.928567:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584542325.928569:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 24125 00000100:00080000:18.0:1584542325.928571:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584542325.928579:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542325.928581:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542325.928581:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542325.928582:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542325.928583:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542325.928584:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542325.928585:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542325.928586:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542325.934715:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542325.934719:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542325.934720:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542325.934721:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542325.934723:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542325.934724:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542325.934726:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542325.934728:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542406 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584542325.934757:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078469136 new x1661492078469168 00000800:00020000:2.0:1584542331.916077:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542331.916079:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542331.916081:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542331.916461:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584542331.928514:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542325/real 1584542325] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542412 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584542331.928526:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584542331.928527:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584542331.928529:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584542331.928535:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542331.928537:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542331.928537:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542331.928539:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542331.928539:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542331.928541:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584542331.928541:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584542331.928543:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 24131 00000100:00080000:18.0:1584542331.928545:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584542331.928554:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542331.928555:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542331.928555:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542331.928557:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542331.928557:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542331.928558:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542331.928559:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542331.928560:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542331.934754:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542331.934758:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542331.934760:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542331.934761:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542331.934762:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542331.934764:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542331.934766:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542331.934767:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542412 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584542331.934797:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078469168 new x1661492078469216 00000800:00020000:2.0:1584542337.916032:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542337.916035:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542337.916036:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542337.916473:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584542337.928528:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542331/real 1584542331] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542418 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584542337.928535:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584542337.928536:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584542337.928538:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584542337.928543:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542337.928544:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542337.928545:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542337.928546:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542337.928547:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542337.928548:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584542337.928549:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584542337.928550:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 24137 00000100:00080000:18.0:1584542337.928552:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584542337.928561:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542337.928562:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542337.928563:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542337.928564:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542337.928564:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542337.928566:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542337.928567:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542337.928568:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:22.0:1584542337.964400:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:22.0:1584542337.964404:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:22.0:1584542337.964405:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:22.0:1584542337.964406:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:22.0:1584542337.964408:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:22.0:1584542337.964409:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:22.0:1584542337.964411:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:22.0:1584542337.964413:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542418 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584542337.964443:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078469216 new x1661492078469248 00000800:00020000:2.0:1584542343.916036:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542343.916039:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542343.916042:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542343.916498:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584542343.928611:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542337/real 1584542337] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542424 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584542343.928622:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584542343.928623:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584542343.928626:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584542343.928647:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542343.928649:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0000_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542343.928652:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542343.928654:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0002_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542343.928654:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0003_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542343.928656:0:4066:0:(recover.c:58:ptlrpc_initiate_recovery()) public1-OST0003_UUID: starting recovery 00000100:00080000:18.0:1584542343.928657:0:4066:0:(import.c:684:ptlrpc_connect_import()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:18.0:1584542343.928659:0:4066:0:(import.c:528:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to NID 10.10.2.22@o2ib last attempt 24143 00000100:00080000:18.0:1584542343.928663:0:4066:0:(import.c:604:import_select_connection()) public1-OST0003-osc-ffff8d3ae8ab1800: import ffff8d31d6e0f000 using connection 10.10.2.22@o2ib/10.10.2.22@o2ib 00000100:00080000:18.0:1584542343.928671:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542343.928673:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0004_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542343.928674:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542343.928675:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0005_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542343.928676:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542343.928678:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0006_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:18.0:1584542343.928678:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:18.0:1584542343.928680:0:4066:0:(pinger.c:261:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->public1-OST0007_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1584542343.934754:0:25724:0:(import.c:1047:ptlrpc_connect_interpret()) public1-OST0003-osc-ffff8d3ae8ab1800: connect to target with instance 5 00000080:00080000:2.0:1584542343.934761:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:00080000:2.0:1584542343.934764:0:25724:0:(import.c:1218:ptlrpc_connect_interpret()) reconnected to public1-OST0003_UUID@10.10.2.22@o2ib after partition 00000100:00080000:2.0:1584542343.934765:0:25724:0:(import.c:1236:ptlrpc_connect_interpret()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1584542343.934767:0:25724:0:(import.c:1549:ptlrpc_import_recovery_state_machine()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from RECOVER to FULL 00000080:00080000:2.0:1584542343.934771:0:25724:0:(lcommon_misc.c:70:cl_init_ea_size()) updating def/max_easize: 72/240 00000100:02000000:2.0:1584542343.934773:0:25724:0:(import.c:1555:ptlrpc_import_recovery_state_machine()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection restored to 10.10.2.22@o2ib (at 10.10.2.22@o2ib) 00000100:00080000:2.0:1584542343.934775:0:25724:0:(recover.c:223:ptlrpc_wake_delayed()) @@@ waking (set ffff8d32ef727e80): req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542424 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:00080000:18.0:1584542343.934809:0:25727:0:(client.c:3320:ptlrpc_set_bulk_mbits()) resend bulk old x1661492078469248 new x1661492078469328 00000800:00020000:2.0:1584542349.916064:0:25711:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 00000800:00020000:2.0:1584542349.916067:0:25711:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.10.2.22@o2ib (5): c: 8, oc: 0, rc: 8 00000800:00000100:2.0:1584542349.916069:0:25711:0:(o2iblnd_cb.c:2065:kiblnd_close_conn_locked()) Closing conn to 10.10.2.22@o2ib: error -110(waiting) 00000100:00020000:0.0:1584542349.916492:0:25711:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8d31d0b23a00 00000100:00000400:18.0:1584542349.928578:0:25727:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584542343/real 1584542343] req@ffff8d3396b56780 x1661492078327264/t0(0) o3->public1-OST0003-osc-ffff8d3ae8ab1800@10.10.2.22@o2ib:6/4 lens 488/440 e 0 to 1 dl 1584542430 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 00000100:02000400:18.0:1584542349.928588:0:25727:0:(import.c:179:ptlrpc_set_import_discon()) public1-OST0003-osc-ffff8d3ae8ab1800: Connection to public1-OST0003 (at 10.10.2.22@o2ib) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:18.0:1584542349.928590:0:25727:0:(import.c:187:ptlrpc_set_import_discon()) ffff8d31d6e0f000 public1-OST0003_UUID: changing import state from FULL to DISCONN 00000100:00080000:18.0:1584542349.928591:0:25727:0:(import.c:422:ptlrpc_pinger_force()) public1-OST0003_UUID: waking up pinger s:DISCONN 00000100:00080000:18.0:1584542349.928599:0:4066:0:(pinger.c:248:ptlrpc_pinger_process_import()) eddc1d5f-0847-ea0c-2573-7f8e65a9b5dc->